Sep 16 12:46:05.947846 ip-10-0-141-139 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 12:46:05.947857 ip-10-0-141-139 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 12:46:05.947864 ip-10-0-141-139 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 12:46:05.948042 ip-10-0-141-139 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 12:46:16.197110 ip-10-0-141-139 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 12:46:16.197130 ip-10-0-141-139 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3db519cecab149c4a97e5350179a03cb -- Sep 16 12:48:12.685463 ip-10-0-141-139 systemd[1]: Starting Kubernetes Kubelet... Sep 16 12:48:13.109671 ip-10-0-141-139 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 12:48:13.109671 ip-10-0-141-139 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 12:48:13.109671 ip-10-0-141-139 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 12:48:13.109671 ip-10-0-141-139 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 12:48:13.109671 ip-10-0-141-139 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 12:48:13.111209 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.111112 2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 12:48:13.115758 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115737 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 12:48:13.115758 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115755 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 12:48:13.115758 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115759 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 12:48:13.115758 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115763 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115766 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115770 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115774 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115776 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115779 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115782 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115785 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115788 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115790 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115793 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115796 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115798 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115801 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115803 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115807 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115810 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115812 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115815 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115818 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 12:48:13.115920 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115821 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115823 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115831 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115834 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115836 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115839 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115842 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115844 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115847 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115849 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115852 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115854 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115858 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115862 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115866 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115870 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115874 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115876 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115880 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 12:48:13.116406 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115883 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115886 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115888 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115891 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115893 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115896 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115899 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115902 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115904 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115907 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115910 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115913 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115915 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115918 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115921 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115923 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115926 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115929 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115932 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115934 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 12:48:13.116867 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115937 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115940 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115942 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115945 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115947 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115950 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115953 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115956 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115959 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115962 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115964 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115967 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115969 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115972 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115974 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115977 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115980 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115983 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115985 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115988 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 12:48:13.117447 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115990 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115995 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.115998 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116000 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116003 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116006 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116008 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116011 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116014 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116016 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116019 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116021 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116024 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116026 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116030 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116034 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116037 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116720 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116727 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116730 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 12:48:13.117946 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116734 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116736 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116739 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116744 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116748 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116751 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116754 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116757 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116760 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116763 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116766 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116768 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116771 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116774 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116777 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116780 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116783 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116785 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116788 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 12:48:13.118445 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116790 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116793 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116796 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116798 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116801 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116803 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116806 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116809 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116812 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116815 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116817 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116821 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116825 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116828 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116830 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116833 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116836 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116840 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116843 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116845 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 12:48:13.118948 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116848 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116851 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116853 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116856 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116859 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116862 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116865 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116868 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116871 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116874 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116877 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116879 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116882 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116885 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116888 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116891 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116894 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116896 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116899 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 12:48:13.119463 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116902 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116905 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116907 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116910 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116912 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116915 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116918 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116921 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116924 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116927 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116929 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116932 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116936 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116939 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116942 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116945 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116948 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116952 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116954 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 12:48:13.119947 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116957 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116960 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116962 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116966 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116968 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116971 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116973 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116976 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116979 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116982 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116984 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116987 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116990 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116992 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116995 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.116998 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117000 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117003 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117006 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117384 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 12:48:13.120434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117390 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117393 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117395 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117399 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117401 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117404 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117407 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117409 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117412 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117414 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117418 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117422 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117425 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117428 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117431 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117434 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117437 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117439 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117442 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117445 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 12:48:13.120914 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117448 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117451 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117454 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117457 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117460 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117462 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117465 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117467 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117470 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117472 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117475 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117478 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117480 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117483 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117486 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117488 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117492 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117494 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117497 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117499 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 12:48:13.121461 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117502 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117504 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117507 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117509 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117512 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117514 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117517 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117520 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117522 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117525 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117527 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117530 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117533 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117535 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117538 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117540 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117543 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117545 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117548 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117550 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 12:48:13.121945 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117552 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117555 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117558 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117562 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117566 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117569 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117573 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117575 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117578 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117580 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117583 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117586 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117588 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117590 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117593 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117596 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117600 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117602 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117612 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117615 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 12:48:13.122434 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117617 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117620 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117623 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117625 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117628 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117630 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117633 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117635 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117638 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117640 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117643 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117645 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117648 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117650 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117653 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117656 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117658 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.117662 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117732 2584 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117767 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 12:48:13.123001 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117774 2584 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117779 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117783 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117786 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117790 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117794 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117798 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117801 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117806 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117810 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117813 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117816 2584 flags.go:64] FLAG: --cgroup-root="" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117819 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117822 2584 flags.go:64] FLAG: --client-ca-file="" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117826 2584 flags.go:64] FLAG: --cloud-provider="external" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117829 2584 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117833 2584 flags.go:64] FLAG: --cluster-domain="" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117836 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117839 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117842 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117846 2584 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117849 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117852 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 12:48:13.123523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117856 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117859 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117862 2584 flags.go:64] FLAG: --contention-profiling="false" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117865 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117869 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117872 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117875 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117879 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117882 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117885 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117888 2584 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117891 2584 flags.go:64] FLAG: --enable-server="true" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117894 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117898 2584 flags.go:64] FLAG: --event-burst="100" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117901 2584 flags.go:64] FLAG: --event-qps="50" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117904 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117907 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117911 2584 flags.go:64] FLAG: --eviction-hard="" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117915 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117918 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117921 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117924 2584 flags.go:64] FLAG: --eviction-soft="" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117927 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117930 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117933 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 12:48:13.124067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117936 2584 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117940 2584 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117942 2584 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117945 2584 flags.go:64] FLAG: --feature-gates="" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117949 2584 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117953 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117956 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117962 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117965 2584 flags.go:64] FLAG: --healthz-port="10248" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117968 2584 flags.go:64] FLAG: --help="false" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117971 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117974 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117977 2584 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117980 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117984 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117987 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117990 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117993 2584 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117996 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.117999 2584 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118002 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118005 2584 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118008 2584 flags.go:64] FLAG: --kube-reserved="" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118011 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 12:48:13.124687 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118015 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118018 2584 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118021 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118024 2584 flags.go:64] FLAG: --lock-file="" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118027 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118030 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118033 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118038 2584 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118042 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118044 2584 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118047 2584 flags.go:64] FLAG: --logging-format="text" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118050 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118054 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118057 2584 flags.go:64] FLAG: --manifest-url="" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118059 2584 flags.go:64] FLAG: --manifest-url-header="" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118069 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118073 2584 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118077 2584 flags.go:64] FLAG: --max-pods="110" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118080 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118083 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118086 2584 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118088 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118092 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118095 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118098 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 12:48:13.125314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118104 2584 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118108 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118111 2584 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118114 2584 flags.go:64] FLAG: --pod-cidr="" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118117 2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118122 2584 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118125 2584 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118130 2584 flags.go:64] FLAG: --pods-per-core="0" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118133 2584 flags.go:64] FLAG: --port="10250" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118136 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118139 2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c579a52094340964" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118142 2584 flags.go:64] FLAG: --qos-reserved="" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118145 2584 flags.go:64] FLAG: --read-only-port="10255" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118148 2584 flags.go:64] FLAG: --register-node="true" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118151 2584 flags.go:64] FLAG: --register-with-taints="" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118155 2584 flags.go:64] FLAG: --registry-burst="10" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118171 2584 flags.go:64] FLAG: --registry-qps="5" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118174 2584 flags.go:64] FLAG: --reserved-cpus="" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118177 2584 flags.go:64] FLAG: --reserved-memory="" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118181 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118184 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118187 2584 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118190 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118194 2584 flags.go:64] FLAG: --runonce="false" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118197 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 12:48:13.125925 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118200 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118203 2584 flags.go:64] FLAG: --seccomp-default="false" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118206 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118209 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118212 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118215 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118218 2584 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118221 2584 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118224 2584 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118227 2584 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118230 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118233 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118236 2584 flags.go:64] FLAG: --system-cgroups="" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118239 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118245 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118248 2584 flags.go:64] FLAG: --tls-cert-file="" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118251 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118256 2584 flags.go:64] FLAG: --tls-min-version="" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118259 2584 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118262 2584 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118265 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118268 2584 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118271 2584 flags.go:64] FLAG: --v="2" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118275 2584 flags.go:64] FLAG: --version="false" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118279 2584 flags.go:64] FLAG: --vmodule="" Sep 16 12:48:13.126569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118283 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.118286 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118399 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118403 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118406 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118411 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118414 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118417 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118420 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118423 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118425 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118428 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118431 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118434 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118436 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118439 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118442 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118445 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118447 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118450 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 12:48:13.127202 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118453 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118457 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118460 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118463 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118465 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118468 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118471 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118473 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118476 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118479 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118481 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118484 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118487 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118490 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118492 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118495 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118498 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118503 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118505 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118508 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 12:48:13.127693 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118510 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118513 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118516 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118518 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118521 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118523 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118526 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118528 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118531 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118534 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118537 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118539 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118542 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118548 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118552 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118555 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118558 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118561 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118564 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 12:48:13.128194 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118566 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118569 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118571 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118574 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118577 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118579 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118583 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118585 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118588 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118590 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118595 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118597 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118600 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118604 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118616 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118620 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118622 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118625 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118628 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 12:48:13.128663 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118630 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118633 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118636 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118638 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118641 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118643 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118646 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118650 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118653 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118655 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118658 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118675 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118679 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118682 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118685 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118688 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118690 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118694 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118697 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 12:48:13.129124 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118700 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118704 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118707 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.118711 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.119244 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.126177 2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.126191 2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.127046 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.127054 2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127109 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127114 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127118 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127121 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127124 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 12:48:13.129598 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127126 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127129 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127132 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127135 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127138 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127140 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127143 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127145 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127148 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127150 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127153 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127155 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127174 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127177 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127179 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127182 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127185 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127188 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127190 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127198 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 12:48:13.129975 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127201 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127204 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127207 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127210 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127213 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127216 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127219 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127222 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127225 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127228 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127232 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127236 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127239 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127242 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127245 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127248 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127250 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127253 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127256 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 12:48:13.130477 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127258 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127261 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127264 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127266 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127269 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127272 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127274 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127277 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127279 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127282 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127285 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127287 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127290 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127293 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127296 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127299 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127302 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127306 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127309 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127311 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 12:48:13.130955 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127314 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127317 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127319 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127322 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127325 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127329 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127332 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127335 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127338 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127341 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127344 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127347 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127350 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127354 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127356 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127359 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127362 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127364 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127367 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 12:48:13.131460 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127369 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127373 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127377 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127379 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127382 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127385 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127388 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127391 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127393 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127396 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127399 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127402 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127405 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127407 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127410 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 12:48:13.131919 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127412 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.127418 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127525 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127529 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127532 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127534 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127538 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127540 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127543 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127547 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127549 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127552 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127554 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127557 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127560 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 12:48:13.132298 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127562 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127565 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127568 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127572 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127576 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127579 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127582 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127584 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127587 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127590 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127594 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127597 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127599 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127602 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127606 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127609 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127612 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127614 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127617 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 12:48:13.132660 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127620 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127622 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127625 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127627 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127630 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127632 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127635 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127637 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127640 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127643 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127645 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127648 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127650 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127653 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127655 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127658 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127661 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127665 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127668 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 12:48:13.133116 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127671 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127674 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127677 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127679 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127682 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127686 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127689 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127692 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127694 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127698 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127701 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127703 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127706 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127709 2584 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127711 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127714 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127717 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127719 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127722 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127724 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 12:48:13.133585 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127727 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127730 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127732 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127735 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127738 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127740 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127743 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127745 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127748 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127751 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127753 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127756 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127759 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127761 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127764 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127767 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127769 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127775 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127778 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127780 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 12:48:13.134061 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127783 2584 feature_gate.go:328] unrecognized feature gate: Example Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127786 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127788 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127791 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127793 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127796 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127798 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:13.127801 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.127806 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.127892 2584 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.129927 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.130925 2584 server.go:1019] "Starting client certificate rotation" Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.131024 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.131826 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 12:48:13.134628 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.133670 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 12:48:13.137413 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.137382 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 12:48:13.147967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.147947 2584 log.go:25] "Validated CRI v1 runtime API" Sep 16 12:48:13.150794 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.150781 2584 log.go:25] "Validated CRI v1 image API" Sep 16 12:48:13.151702 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.151685 2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 12:48:13.157722 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.157698 2584 fs.go:135] Filesystem UUIDs: map[647b8c84-765b-4025-b0b0-f45ae5b9c2bc:/dev/nvme0n1p3 76ce29f4-292e-4972-9b33-18a0a3b284d2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 16 12:48:13.157722 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.157716 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 12:48:13.163790 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.163680 2584 manager.go:216] Machine: {Timestamp:2026-09-16 12:48:13.161249647 +0000 UTC m=+0.353898249 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119714 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dc5c6143cedb89630fac9b2c1cde1 SystemUUID:ec2dc5c6-143c-edb8-9630-fac9b2c1cde1 BootID:3db519ce-cab1-49c4-a97e-5350179a03cb Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:14:9d:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:14:9d:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:8c:c4:88:5f:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 12:48:13.163790 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.163783 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 12:48:13.163960 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.163856 2584 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 12:48:13.164600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.164584 2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 12:48:13.166476 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.166451 2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 12:48:13.166703 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.166478 2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 12:48:13.167911 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.167901 2584 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 12:48:13.167947 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.167915 2584 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 12:48:13.167947 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.167922 2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 12:48:13.168706 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.168696 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 12:48:13.169597 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.169588 2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 12:48:13.173197 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.173186 2584 state_mem.go:36] "Initialized new in-memory state store" Sep 16 12:48:13.173305 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.173296 2584 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 12:48:13.173338 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.173329 2584 kubelet.go:486] "Attempting to sync node with API server" Sep 16 12:48:13.173365 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.173346 2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 12:48:13.173365 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.173363 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 12:48:13.173415 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.173372 2584 kubelet.go:398] "Adding apiserver pod source" Sep 16 12:48:13.173415 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.173380 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 12:48:13.173720 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.173708 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 12:48:13.173760 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.173747 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 12:48:13.176071 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.176055 2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 12:48:13.178491 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.178478 2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 12:48:13.178553 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.178508 2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 12:48:13.180775 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180762 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 12:48:13.180831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180782 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 12:48:13.180831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180788 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 12:48:13.180831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180794 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 12:48:13.180831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180803 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 12:48:13.180831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180808 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 12:48:13.180831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180823 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 12:48:13.180831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180828 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 12:48:13.181006 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180835 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 12:48:13.181006 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180841 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 12:48:13.181006 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180853 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 12:48:13.181006 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.180859 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 12:48:13.181653 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.181643 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 12:48:13.181653 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.181652 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 12:48:13.185011 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.184993 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 12:48:13.185372 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.185359 2584 server.go:1268] "Started kubelet" Sep 16 12:48:13.185451 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.185427 2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 12:48:13.185587 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.185543 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 12:48:13.185696 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.185543 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 12:48:13.185696 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.185617 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 12:48:13.185696 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.185654 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 12:48:13.189426 ip-10-0-141-139 systemd[1]: Started Kubernetes Kubelet. Sep 16 12:48:13.190144 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.189728 2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 12:48:13.190779 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.190760 2584 server.go:310] "Adding debug handlers to kubelet server" Sep 16 12:48:13.192729 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.192462 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bmhx8" Sep 16 12:48:13.194010 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.193989 2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 12:48:13.195914 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.194911 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-139.ec2.internal.18d5cdf84788227a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-139.ec2.internal,UID:ip-10-0-141-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-139.ec2.internal,},FirstTimestamp:2026-09-16 12:48:13.185327738 +0000 UTC m=+0.377976317,LastTimestamp:2026-09-16 12:48:13.185327738 +0000 UTC m=+0.377976317,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-139.ec2.internal,}" Sep 16 12:48:13.199727 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.199713 2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 12:48:13.199727 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.199722 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 12:48:13.199843 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.199796 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:13.200243 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.200223 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bmhx8" Sep 16 12:48:13.200596 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.200568 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 12:48:13.200596 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.200572 2584 factory.go:55] Registering systemd factory Sep 16 12:48:13.200720 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.200597 2584 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 12:48:13.200720 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.200613 2584 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 12:48:13.200720 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.200601 2584 factory.go:223] Registration of the systemd container factory successfully Sep 16 12:48:13.200864 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.200744 2584 reconstruct.go:97] "Volume reconstruction finished" Sep 16 12:48:13.200864 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.200752 2584 reconciler.go:29] "Reconciler: start to sync state" Sep 16 12:48:13.200977 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.200958 2584 factory.go:153] Registering CRI-O factory Sep 16 12:48:13.200977 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.200977 2584 factory.go:223] Registration of the crio container factory successfully Sep 16 12:48:13.201084 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.201021 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 12:48:13.201084 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.201035 2584 factory.go:103] Registering Raw factory Sep 16 12:48:13.201084 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.201045 2584 manager.go:1196] Started watching for new ooms in manager Sep 16 12:48:13.201433 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.201422 2584 manager.go:349] Starting recovery of all containers Sep 16 12:48:13.212285 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.212269 2584 manager.go:354] Recovery completed Sep 16 12:48:13.212898 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.212882 2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 12:48:13.216023 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.216005 2584 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-139.ec2.internal\" not found" node="ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.217569 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.217554 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 12:48:13.220615 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.220470 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory" Sep 16 12:48:13.220679 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.220637 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 12:48:13.220679 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.220651 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID" Sep 16 12:48:13.221150 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.221135 2584 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 12:48:13.221231 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.221150 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 12:48:13.221273 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.221149 2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 12:48:13.221273 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.221257 2584 state_mem.go:36] "Initialized new in-memory state store" Sep 16 12:48:13.222290 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.222276 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory" Sep 16 12:48:13.222358 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.222300 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 12:48:13.222358 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.222310 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID" Sep 16 12:48:13.222560 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.222548 2584 policy_none.go:49] "None policy: Start" Sep 16 12:48:13.222590 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.222566 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 12:48:13.222590 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.222576 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 12:48:13.223746 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.223736 2584 policy_none.go:47] "Start" Sep 16 12:48:13.247695 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.247677 2584 manager.go:337] "Starting Device Plugin manager" Sep 16 12:48:13.247789 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.247718 2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 12:48:13.247789 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.247732 2584 server.go:87] "Starting device plugin registration server" Sep 16 12:48:13.248040 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.248027 2584 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 12:48:13.248088 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.248043 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 12:48:13.248155 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.248132 2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 12:48:13.248249 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.248235 2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 12:48:13.248249 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.248249 2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 12:48:13.248877 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.248855 2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 12:48:13.248971 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.248908 2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:13.266452 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.266436 2584 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2612 into the system container \"/system.slice\": " Sep 16 12:48:13.348241 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.348185 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 12:48:13.349423 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.349396 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory" Sep 16 12:48:13.349590 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.349434 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 12:48:13.349590 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.349445 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID" Sep 16 12:48:13.349590 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.349470 2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.361782 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.361707 2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.361782 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.361730 2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-139.ec2.internal\": node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:13.382825 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.382803 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:13.465463 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.465425 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 12:48:13.466638 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.466623 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 12:48:13.466734 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.466674 2584 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 12:48:13.466734 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.466701 2584 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 12:48:13.466828 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.466797 2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 12:48:13.469426 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.469408 2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 12:48:13.483798 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.483767 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:13.567242 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.567189 2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal"] Sep 16 12:48:13.567377 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.567295 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 12:48:13.568364 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.568337 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory" Sep 16 12:48:13.568459 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.568369 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 12:48:13.568459 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.568380 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID" Sep 16 12:48:13.569785 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.569772 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 12:48:13.569900 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.569886 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.569939 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.569917 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 12:48:13.570564 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.570546 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory" Sep 16 12:48:13.570664 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.570567 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory" Sep 16 12:48:13.570664 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.570575 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 12:48:13.570664 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.570585 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 12:48:13.570664 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.570591 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID" Sep 16 12:48:13.570664 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.570594 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID" Sep 16 12:48:13.571871 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.571855 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.571952 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.571883 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 12:48:13.572535 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.572519 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientMemory" Sep 16 12:48:13.572605 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.572548 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 12:48:13.572605 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.572561 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeHasSufficientPID" Sep 16 12:48:13.584274 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.584259 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:13.595820 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.595799 2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-139.ec2.internal\" not found" node="ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.600269 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.600254 2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-139.ec2.internal\" not found" node="ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.602316 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.602302 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76f1b2a16574ce38927f7f5da97679-config\") pod \"kube-apiserver-proxy-ip-10-0-141-139.ec2.internal\" (UID: \"8b76f1b2a16574ce38927f7f5da97679\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.602378 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.602324 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.602378 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.602341 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.684434 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.684357 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:13.703083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.703053 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.703083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.703084 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.703257 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.703104 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76f1b2a16574ce38927f7f5da97679-config\") pod \"kube-apiserver-proxy-ip-10-0-141-139.ec2.internal\" (UID: \"8b76f1b2a16574ce38927f7f5da97679\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.703257 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.703185 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.703257 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.703168 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b76f1b2a16574ce38927f7f5da97679-config\") pod \"kube-apiserver-proxy-ip-10-0-141-139.ec2.internal\" (UID: \"8b76f1b2a16574ce38927f7f5da97679\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.703257 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.703176 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/628f3dc115ebc1acc6ed7da71915e638-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal\" (UID: \"628f3dc115ebc1acc6ed7da71915e638\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.784627 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.784588 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:13.891266 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.891210 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:13.898996 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.898974 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.903573 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:13.903559 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" Sep 16 12:48:13.992667 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:13.992625 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:14.097617 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:14.097587 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:14.105751 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.105730 2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 12:48:14.131235 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.131211 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 12:48:14.131720 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.131318 2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 12:48:14.131720 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.131359 2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 12:48:14.131720 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.131372 2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 12:48:14.198031 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:14.198006 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:14.204574 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.204542 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 12:43:13 +0000 UTC" deadline="2028-04-28 20:40:47.629992779 +0000 UTC" Sep 16 12:48:14.204574 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.204567 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14167h52m33.425429207s" Sep 16 12:48:14.299062 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:14.298987 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-139.ec2.internal\" not found" Sep 16 12:48:14.303563 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.303548 2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 12:48:14.400623 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.400591 2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" Sep 16 12:48:14.412209 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.412187 2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 12:48:14.413320 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.413307 2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" Sep 16 12:48:14.426613 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.426595 2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 12:48:14.650636 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:14.650602 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628f3dc115ebc1acc6ed7da71915e638.slice/crio-476259fd7c79be0547ec5adf0d1978017c857e2eaf7c7d96bab8e46cca2224c4 WatchSource:0}: Error finding container 476259fd7c79be0547ec5adf0d1978017c857e2eaf7c7d96bab8e46cca2224c4: Status 404 returned error can't find the container with id 476259fd7c79be0547ec5adf0d1978017c857e2eaf7c7d96bab8e46cca2224c4 Sep 16 12:48:14.650995 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:14.650977 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b76f1b2a16574ce38927f7f5da97679.slice/crio-1ca7bbb1c5fdef025932fcebe37e4bbe832a27f96defeaf119e19a75b8792946 WatchSource:0}: Error finding container 1ca7bbb1c5fdef025932fcebe37e4bbe832a27f96defeaf119e19a75b8792946: Status 404 returned error can't find the container with id 1ca7bbb1c5fdef025932fcebe37e4bbe832a27f96defeaf119e19a75b8792946 Sep 16 12:48:14.658963 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:14.658949 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 12:48:15.095491 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.095457 2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 12:48:15.176637 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.176608 2584 apiserver.go:52] "Watching apiserver" Sep 16 12:48:15.186699 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.186672 2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 12:48:15.187105 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.187079 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn","openshift-cluster-node-tuning-operator/tuned-9zltf","openshift-dns/node-resolver-c64wg","openshift-image-registry/node-ca-gbhf2","openshift-multus/multus-g7gz4","openshift-multus/network-metrics-daemon-jjdcd","openshift-ovn-kubernetes/ovnkube-node-wzmzs","kube-system/global-pull-secret-syncer-hrjnz","kube-system/konnectivity-agent-fgbqm","kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal","openshift-multus/multus-additional-cni-plugins-bslcs","openshift-network-diagnostics/network-check-target-btfcd","openshift-network-operator/iptables-alerter-2259d"] Sep 16 12:48:15.188768 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.188742 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.190152 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.190130 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:15.190277 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.190250 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:15.191939 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.191504 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.191939 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.191632 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 12:48:15.191939 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.191840 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 12:48:15.192688 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.192669 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.193223 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.193201 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tw7g\"" Sep 16 12:48:15.193344 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.193296 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 12:48:15.193414 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.193366 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 12:48:15.193557 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.193538 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 12:48:15.193664 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.193545 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 12:48:15.194445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.194327 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zq9w2\"" Sep 16 12:48:15.194445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.194334 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 12:48:15.194445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.194388 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.194445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.194417 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 12:48:15.194967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.194946 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 12:48:15.195371 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.195345 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlh26\"" Sep 16 12:48:15.196116 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.196094 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:15.196233 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.196167 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:15.196534 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.196517 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 12:48:15.196616 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.196572 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 12:48:15.197694 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.197258 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 12:48:15.197694 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.197329 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6qbgp\"" Sep 16 12:48:15.197694 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.197381 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 12:48:15.197694 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.197518 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 12:48:15.198458 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.197768 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:15.198517 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.198191 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 12:48:15.199589 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.199510 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.201278 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.201219 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 12:48:15.202070 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.202040 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wtwnh\"" Sep 16 12:48:15.202430 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.202408 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 12:48:15.202802 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.202785 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 12:48:15.203134 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.203114 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 12:48:15.203299 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.203283 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 12:48:15.203547 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.203519 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.203649 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.203625 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9222l\"" Sep 16 12:48:15.207055 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.206753 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.207055 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.206780 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 12:48:15.208215 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.208172 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vxvw8\"" Sep 16 12:48:15.208371 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.208224 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 12:48:15.208371 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.208243 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 12:48:15.208831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.208814 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:15.208933 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.208877 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:15.209062 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209038 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.209149 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209071 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6555a32-a5f6-4d65-82c7-6aaabaff1fbc-host\") pod \"node-ca-gbhf2\" (UID: \"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc\") " pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.209149 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209092 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-system-cni-dir\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.209149 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209109 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b868f668-a89e-43dd-bb29-18c856372b90-cni-binary-copy\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.209149 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209125 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-hostroot\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.209149 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209139 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtnp\" (UniqueName: \"kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp\") pod \"network-check-target-btfcd\" (UID: \"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04\") " pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:15.209439 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209160 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-var-lib-cni-multus\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.209439 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209205 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dda716d8-6057-42d2-8a6f-ee5cddd8592a-tmp\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.209439 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209233 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-run-openvswitch\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.209439 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209255 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-systemd\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.209439 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209270 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-multus-cni-dir\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.209439 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209284 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-run-netns\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.209439 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209304 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-var-lib-openvswitch\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.209439 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209328 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-node-log\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.209439 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209355 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:15.209439 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209419 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37248adf-9ac8-4fd6-a693-daf752b60555-konnectivity-ca\") pod \"konnectivity-agent-fgbqm\" (UID: \"37248adf-9ac8-4fd6-a693-daf752b60555\") " pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209469 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-device-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209500 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-tuned\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209524 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-slash\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209551 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-etc-openvswitch\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209578 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209628 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-socket-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209650 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209688 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209747 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209654 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-sysctl-conf\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209800 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209843 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-ovn-node-metrics-cert\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209870 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzww\" (UniqueName: \"kubernetes.io/projected/9711ead1-9753-4be1-b0f5-78cd0ac6f89f-kube-api-access-zwzww\") pod \"node-resolver-c64wg\" (UID: \"9711ead1-9753-4be1-b0f5-78cd0ac6f89f\") " pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209909 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-env-overrides\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209941 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-var-lib-cni-bin\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.209985 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.209969 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b868f668-a89e-43dd-bb29-18c856372b90-multus-daemon-config\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210009 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-lib-modules\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210021 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-b49k8\"" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210037 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-var-lib-kubelet\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210063 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-run-ovn\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210088 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-ovnkube-script-lib\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210127 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-multus-socket-dir-parent\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210178 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-modprobe-d\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210204 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-systemd-units\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210232 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-log-socket\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210256 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84kt\" (UniqueName: \"kubernetes.io/projected/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-kube-api-access-w84kt\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210281 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-cni-netd\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210307 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-cnibin\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210330 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-etc-kubernetes\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210357 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6555a32-a5f6-4d65-82c7-6aaabaff1fbc-serviceca\") pod \"node-ca-gbhf2\" (UID: \"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc\") " pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210396 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-var-lib-kubelet\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210438 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-kubelet\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.210831 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210496 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9711ead1-9753-4be1-b0f5-78cd0ac6f89f-hosts-file\") pod \"node-resolver-c64wg\" (UID: \"9711ead1-9753-4be1-b0f5-78cd0ac6f89f\") " pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210535 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-kubernetes\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210568 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210566 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-run\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210678 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjgr\" (UniqueName: \"kubernetes.io/projected/418a1352-4779-435a-880c-627e3480b111-kube-api-access-9hjgr\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210721 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-sys-fs\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210768 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-run-k8s-cni-cncf-io\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210804 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnqh\" (UniqueName: \"kubernetes.io/projected/b868f668-a89e-43dd-bb29-18c856372b90-kube-api-access-ghnqh\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210829 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-run-netns\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210853 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210881 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pdh\" (UniqueName: \"kubernetes.io/projected/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-kube-api-access-w8pdh\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210908 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-multus-conf-dir\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.210946 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-ovnkube-config\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211013 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37248adf-9ac8-4fd6-a693-daf752b60555-agent-certs\") pod \"konnectivity-agent-fgbqm\" (UID: \"37248adf-9ac8-4fd6-a693-daf752b60555\") " pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211057 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-registration-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211094 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9711ead1-9753-4be1-b0f5-78cd0ac6f89f-tmp-dir\") pod \"node-resolver-c64wg\" (UID: \"9711ead1-9753-4be1-b0f5-78cd0ac6f89f\") " pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211118 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-sysconfig\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.211715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211138 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-host\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.212426 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211161 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6gn\" (UniqueName: \"kubernetes.io/projected/dda716d8-6057-42d2-8a6f-ee5cddd8592a-kube-api-access-rp6gn\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.212426 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211195 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-run-systemd\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.212426 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211236 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxh7p\" (UniqueName: \"kubernetes.io/projected/b6555a32-a5f6-4d65-82c7-6aaabaff1fbc-kube-api-access-mxh7p\") pod \"node-ca-gbhf2\" (UID: \"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc\") " pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.212426 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211260 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-run-multus-certs\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.212426 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211280 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-sysctl-d\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.212426 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211311 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-sys\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.212426 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.211337 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-cni-bin\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.213434 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.213081 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 12:48:15.213434 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.213157 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rj655\"" Sep 16 12:48:15.213434 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.213241 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 12:48:15.214991 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.214799 2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 12:48:15.242211 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.242190 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ghzlh" Sep 16 12:48:15.251075 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.251049 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ghzlh" Sep 16 12:48:15.302247 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.302225 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 12:48:15.312024 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.311996 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-multus-conf-dir\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.312024 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.311944 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-multus-conf-dir\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.312200 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312065 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-ovnkube-config\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.312200 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312093 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37248adf-9ac8-4fd6-a693-daf752b60555-agent-certs\") pod \"konnectivity-agent-fgbqm\" (UID: \"37248adf-9ac8-4fd6-a693-daf752b60555\") " pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:15.312200 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312119 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-registration-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.312200 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312139 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9711ead1-9753-4be1-b0f5-78cd0ac6f89f-tmp-dir\") pod \"node-resolver-c64wg\" (UID: \"9711ead1-9753-4be1-b0f5-78cd0ac6f89f\") " pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.312200 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312163 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-sysconfig\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312202 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-host\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312231 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6gn\" (UniqueName: \"kubernetes.io/projected/dda716d8-6057-42d2-8a6f-ee5cddd8592a-kube-api-access-rp6gn\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312245 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-run-systemd\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312271 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxh7p\" (UniqueName: \"kubernetes.io/projected/b6555a32-a5f6-4d65-82c7-6aaabaff1fbc-kube-api-access-mxh7p\") pod \"node-ca-gbhf2\" (UID: \"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc\") " pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312306 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-run-multus-certs\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312332 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-sysctl-d\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312351 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-sys\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312371 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-cni-bin\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312396 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312410 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6555a32-a5f6-4d65-82c7-6aaabaff1fbc-host\") pod \"node-ca-gbhf2\" (UID: \"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc\") " pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312424 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-system-cni-dir\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.312467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312449 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b868f668-a89e-43dd-bb29-18c856372b90-cni-binary-copy\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312476 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-hostroot\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312494 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtnp\" (UniqueName: \"kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp\") pod \"network-check-target-btfcd\" (UID: \"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04\") " pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312512 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-var-lib-cni-multus\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312585 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-var-lib-cni-multus\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312588 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-hostroot\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312606 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-run-multus-certs\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312644 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-cni-bin\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312646 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-sys\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312687 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dda716d8-6057-42d2-8a6f-ee5cddd8592a-tmp\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312714 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-run-openvswitch\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312720 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312741 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rns9g\" (UniqueName: \"kubernetes.io/projected/4f90e3ec-14f4-443f-b93a-1a1d05265a71-kube-api-access-rns9g\") pod \"iptables-alerter-2259d\" (UID: \"4f90e3ec-14f4-443f-b93a-1a1d05265a71\") " pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312767 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-registration-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312784 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312793 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-sysctl-d\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312815 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cnl\" (UniqueName: \"kubernetes.io/projected/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-kube-api-access-c5cnl\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.313083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312824 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-ovnkube-config\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312861 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-systemd\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312841 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-run-openvswitch\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312928 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-system-cni-dir\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312947 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-multus-cni-dir\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312974 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-run-netns\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.312988 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-host\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313001 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-var-lib-openvswitch\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313027 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-node-log\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313018 2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313054 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313087 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9711ead1-9753-4be1-b0f5-78cd0ac6f89f-tmp-dir\") pod \"node-resolver-c64wg\" (UID: \"9711ead1-9753-4be1-b0f5-78cd0ac6f89f\") " pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313103 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-sysconfig\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313140 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-run-netns\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313152 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-node-log\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313157 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37248adf-9ac8-4fd6-a693-daf752b60555-konnectivity-ca\") pod \"konnectivity-agent-fgbqm\" (UID: \"37248adf-9ac8-4fd6-a693-daf752b60555\") " pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313180 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-var-lib-openvswitch\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313188 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-device-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.314044 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313027 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-run-systemd\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313213 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-tuned\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.313249 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313254 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-slash\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313260 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-multus-cni-dir\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313279 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b868f668-a89e-43dd-bb29-18c856372b90-cni-binary-copy\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313291 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-etc-openvswitch\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313152 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-systemd\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313340 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6555a32-a5f6-4d65-82c7-6aaabaff1fbc-host\") pod \"node-ca-gbhf2\" (UID: \"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc\") " pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.313368 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs podName:418a1352-4779-435a-880c-627e3480b111 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:15.81334303 +0000 UTC m=+3.005991625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs") pod "network-metrics-daemon-jjdcd" (UID: "418a1352-4779-435a-880c-627e3480b111") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313354 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-device-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313394 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313426 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-slash\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313427 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-socket-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313478 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-sysctl-conf\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313509 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313534 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-socket-dir\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.314981 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313541 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-ovn-node-metrics-cert\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313568 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzww\" (UniqueName: \"kubernetes.io/projected/9711ead1-9753-4be1-b0f5-78cd0ac6f89f-kube-api-access-zwzww\") pod \"node-resolver-c64wg\" (UID: \"9711ead1-9753-4be1-b0f5-78cd0ac6f89f\") " pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313571 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313588 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00648523-fd55-498f-b8c6-704f6c9d7e47-dbus\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313605 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-env-overrides\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313805 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-var-lib-cni-bin\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313840 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b868f668-a89e-43dd-bb29-18c856372b90-multus-daemon-config\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313867 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-lib-modules\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313900 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-var-lib-kubelet\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313927 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-run-ovn\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313957 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313986 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-ovnkube-script-lib\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314014 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-multus-socket-dir-parent\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314040 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-modprobe-d\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314063 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-systemd-units\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314090 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-log-socket\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314119 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.315967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314148 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84kt\" (UniqueName: \"kubernetes.io/projected/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-kube-api-access-w84kt\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314175 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-cni-netd\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314252 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-cnibin\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314280 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-etc-kubernetes\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314306 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6555a32-a5f6-4d65-82c7-6aaabaff1fbc-serviceca\") pod \"node-ca-gbhf2\" (UID: \"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc\") " pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314352 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-var-lib-kubelet\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314380 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-kubelet\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314384 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-env-overrides\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314409 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9711ead1-9753-4be1-b0f5-78cd0ac6f89f-hosts-file\") pod \"node-resolver-c64wg\" (UID: \"9711ead1-9753-4be1-b0f5-78cd0ac6f89f\") " pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314442 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f90e3ec-14f4-443f-b93a-1a1d05265a71-host-slash\") pod \"iptables-alerter-2259d\" (UID: \"4f90e3ec-14f4-443f-b93a-1a1d05265a71\") " pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314470 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-system-cni-dir\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314480 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9711ead1-9753-4be1-b0f5-78cd0ac6f89f-hosts-file\") pod \"node-resolver-c64wg\" (UID: \"9711ead1-9753-4be1-b0f5-78cd0ac6f89f\") " pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314493 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314520 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-kubernetes\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314542 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-run\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314543 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-multus-socket-dir-parent\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314567 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjgr\" (UniqueName: \"kubernetes.io/projected/418a1352-4779-435a-880c-627e3480b111-kube-api-access-9hjgr\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:15.316636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314590 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-sys-fs\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314613 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00648523-fd55-498f-b8c6-704f6c9d7e47-kubelet-config\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314631 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-modprobe-d\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.313396 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-etc-openvswitch\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314637 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-cnibin\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314673 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-ovnkube-script-lib\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314685 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314715 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-run-k8s-cni-cncf-io\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314684 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-systemd-units\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314195 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37248adf-9ac8-4fd6-a693-daf752b60555-konnectivity-ca\") pod \"konnectivity-agent-fgbqm\" (UID: \"37248adf-9ac8-4fd6-a693-daf752b60555\") " pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314940 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-var-lib-kubelet\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314963 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-sysctl-conf\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314963 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-log-socket\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.314979 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315016 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-etc-kubernetes\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315049 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-lib-modules\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315056 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-var-lib-kubelet\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.317184 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315057 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-run-k8s-cni-cncf-io\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315105 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-run\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315115 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-host-var-lib-cni-bin\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315161 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-run-ovn\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315154 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnqh\" (UniqueName: \"kubernetes.io/projected/b868f668-a89e-43dd-bb29-18c856372b90-kube-api-access-ghnqh\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315232 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-run-netns\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315263 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315291 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pdh\" (UniqueName: \"kubernetes.io/projected/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-kube-api-access-w8pdh\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315322 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f90e3ec-14f4-443f-b93a-1a1d05265a71-iptables-alerter-script\") pod \"iptables-alerter-2259d\" (UID: \"4f90e3ec-14f4-443f-b93a-1a1d05265a71\") " pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315418 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6555a32-a5f6-4d65-82c7-6aaabaff1fbc-serviceca\") pod \"node-ca-gbhf2\" (UID: \"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc\") " pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315464 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-cni-netd\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315506 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-kubelet\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315552 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b868f668-a89e-43dd-bb29-18c856372b90-cnibin\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315570 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b868f668-a89e-43dd-bb29-18c856372b90-multus-daemon-config\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315602 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-host-run-netns\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315610 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-sys-fs\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315653 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-kubernetes\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.315696 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.317998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.317518 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-ovn-node-metrics-cert\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.318899 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.317672 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dda716d8-6057-42d2-8a6f-ee5cddd8592a-tmp\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.318899 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.317721 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dda716d8-6057-42d2-8a6f-ee5cddd8592a-etc-tuned\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.319659 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.319597 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37248adf-9ac8-4fd6-a693-daf752b60555-agent-certs\") pod \"konnectivity-agent-fgbqm\" (UID: \"37248adf-9ac8-4fd6-a693-daf752b60555\") " pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:15.326684 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.326660 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 12:48:15.326786 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.326690 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 12:48:15.326786 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.326705 2584 projected.go:196] Error preparing data for projected volume kube-api-access-2dtnp for pod openshift-network-diagnostics/network-check-target-btfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:15.326786 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.326765 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp podName:aacaaf0a-8a6f-4ced-9c88-1fab1496cb04 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:15.826747707 +0000 UTC m=+3.019396300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2dtnp" (UniqueName: "kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp") pod "network-check-target-btfcd" (UID: "aacaaf0a-8a6f-4ced-9c88-1fab1496cb04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:15.327423 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.327386 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pdh\" (UniqueName: \"kubernetes.io/projected/3b18e641-43cd-4ad1-b68e-63d3427fc9fb-kube-api-access-w8pdh\") pod \"aws-ebs-csi-driver-node-46szn\" (UID: \"3b18e641-43cd-4ad1-b68e-63d3427fc9fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.328055 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.328000 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjgr\" (UniqueName: \"kubernetes.io/projected/418a1352-4779-435a-880c-627e3480b111-kube-api-access-9hjgr\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:15.328480 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.328454 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzww\" (UniqueName: \"kubernetes.io/projected/9711ead1-9753-4be1-b0f5-78cd0ac6f89f-kube-api-access-zwzww\") pod \"node-resolver-c64wg\" (UID: \"9711ead1-9753-4be1-b0f5-78cd0ac6f89f\") " pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.328567 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.328492 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnqh\" (UniqueName: \"kubernetes.io/projected/b868f668-a89e-43dd-bb29-18c856372b90-kube-api-access-ghnqh\") pod \"multus-g7gz4\" (UID: \"b868f668-a89e-43dd-bb29-18c856372b90\") " pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.329185 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.329159 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84kt\" (UniqueName: \"kubernetes.io/projected/10c6c3a7-6ac2-45ea-9726-28260b38b6d8-kube-api-access-w84kt\") pod \"ovnkube-node-wzmzs\" (UID: \"10c6c3a7-6ac2-45ea-9726-28260b38b6d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.329305 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.329276 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6gn\" (UniqueName: \"kubernetes.io/projected/dda716d8-6057-42d2-8a6f-ee5cddd8592a-kube-api-access-rp6gn\") pod \"tuned-9zltf\" (UID: \"dda716d8-6057-42d2-8a6f-ee5cddd8592a\") " pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.330362 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.330344 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxh7p\" (UniqueName: \"kubernetes.io/projected/b6555a32-a5f6-4d65-82c7-6aaabaff1fbc-kube-api-access-mxh7p\") pod \"node-ca-gbhf2\" (UID: \"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc\") " pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.416520 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416442 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00648523-fd55-498f-b8c6-704f6c9d7e47-kubelet-config\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:15.416520 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416481 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-cnibin\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.416520 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416507 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.416777 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416538 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f90e3ec-14f4-443f-b93a-1a1d05265a71-iptables-alerter-script\") pod \"iptables-alerter-2259d\" (UID: \"4f90e3ec-14f4-443f-b93a-1a1d05265a71\") " pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.416777 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416580 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00648523-fd55-498f-b8c6-704f6c9d7e47-kubelet-config\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:15.416777 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416601 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rns9g\" (UniqueName: \"kubernetes.io/projected/4f90e3ec-14f4-443f-b93a-1a1d05265a71-kube-api-access-rns9g\") pod \"iptables-alerter-2259d\" (UID: \"4f90e3ec-14f4-443f-b93a-1a1d05265a71\") " pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.416777 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416584 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-cnibin\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.416777 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416629 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:15.416777 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416657 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cnl\" (UniqueName: \"kubernetes.io/projected/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-kube-api-access-c5cnl\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.416777 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416709 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00648523-fd55-498f-b8c6-704f6c9d7e47-dbus\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:15.416777 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416742 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.417290 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416787 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.417290 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416822 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f90e3ec-14f4-443f-b93a-1a1d05265a71-host-slash\") pod \"iptables-alerter-2259d\" (UID: \"4f90e3ec-14f4-443f-b93a-1a1d05265a71\") " pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.417290 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416848 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-system-cni-dir\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.417290 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.416872 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.417290 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.417079 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f90e3ec-14f4-443f-b93a-1a1d05265a71-iptables-alerter-script\") pod \"iptables-alerter-2259d\" (UID: \"4f90e3ec-14f4-443f-b93a-1a1d05265a71\") " pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.417290 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.417196 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.417290 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.417272 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.417663 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.417316 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00648523-fd55-498f-b8c6-704f6c9d7e47-dbus\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:15.417663 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.417336 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f90e3ec-14f4-443f-b93a-1a1d05265a71-host-slash\") pod \"iptables-alerter-2259d\" (UID: \"4f90e3ec-14f4-443f-b93a-1a1d05265a71\") " pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.417663 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.417384 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-system-cni-dir\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.417663 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.417421 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.417663 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.417489 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:15.417663 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.417545 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret podName:00648523-fd55-498f-b8c6-704f6c9d7e47 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:15.91752734 +0000 UTC m=+3.110175925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret") pod "global-pull-secret-syncer-hrjnz" (UID: "00648523-fd55-498f-b8c6-704f6c9d7e47") : object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:15.417982 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.417909 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.434165 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.434142 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cnl\" (UniqueName: \"kubernetes.io/projected/be40f43f-3d2a-4499-8e88-b78cd35bc9ed-kube-api-access-c5cnl\") pod \"multus-additional-cni-plugins-bslcs\" (UID: \"be40f43f-3d2a-4499-8e88-b78cd35bc9ed\") " pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.447762 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.447742 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rns9g\" (UniqueName: \"kubernetes.io/projected/4f90e3ec-14f4-443f-b93a-1a1d05265a71-kube-api-access-rns9g\") pod \"iptables-alerter-2259d\" (UID: \"4f90e3ec-14f4-443f-b93a-1a1d05265a71\") " pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.471905 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.471865 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" event={"ID":"628f3dc115ebc1acc6ed7da71915e638","Type":"ContainerStarted","Data":"476259fd7c79be0547ec5adf0d1978017c857e2eaf7c7d96bab8e46cca2224c4"} Sep 16 12:48:15.473210 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.473157 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" event={"ID":"8b76f1b2a16574ce38927f7f5da97679","Type":"ContainerStarted","Data":"1ca7bbb1c5fdef025932fcebe37e4bbe832a27f96defeaf119e19a75b8792946"} Sep 16 12:48:15.499287 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.499255 2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 12:48:15.505136 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.505110 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:15.511714 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.511507 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c64wg" Sep 16 12:48:15.517610 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:15.517584 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6c3a7_6ac2_45ea_9726_28260b38b6d8.slice/crio-d75af67e95333bd073ba587847c96cb7e599e7df9f4d98e460a0dfda8236a19a WatchSource:0}: Error finding container d75af67e95333bd073ba587847c96cb7e599e7df9f4d98e460a0dfda8236a19a: Status 404 returned error can't find the container with id d75af67e95333bd073ba587847c96cb7e599e7df9f4d98e460a0dfda8236a19a Sep 16 12:48:15.521671 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.521644 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gbhf2" Sep 16 12:48:15.523444 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:15.523420 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9711ead1_9753_4be1_b0f5_78cd0ac6f89f.slice/crio-514cca67befc9d08e1133c2393ca83f0982871b676847947b07cb7cf32b58b13 WatchSource:0}: Error finding container 514cca67befc9d08e1133c2393ca83f0982871b676847947b07cb7cf32b58b13: Status 404 returned error can't find the container with id 514cca67befc9d08e1133c2393ca83f0982871b676847947b07cb7cf32b58b13 Sep 16 12:48:15.527155 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.527133 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g7gz4" Sep 16 12:48:15.532951 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:15.532911 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6555a32_a5f6_4d65_82c7_6aaabaff1fbc.slice/crio-3d7c3d361e914cf7bb00f43b86a2232bf896098b9496b4581485bc9b7e14f92d WatchSource:0}: Error finding container 3d7c3d361e914cf7bb00f43b86a2232bf896098b9496b4581485bc9b7e14f92d: Status 404 returned error can't find the container with id 3d7c3d361e914cf7bb00f43b86a2232bf896098b9496b4581485bc9b7e14f92d Sep 16 12:48:15.537889 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.537867 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:15.542986 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.542283 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" Sep 16 12:48:15.542986 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:15.542888 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb868f668_a89e_43dd_bb29_18c856372b90.slice/crio-0d9ae068ac8297dbbf5386bcbecd7a6c50a63e03d3d592160212f78ff77df7d4 WatchSource:0}: Error finding container 0d9ae068ac8297dbbf5386bcbecd7a6c50a63e03d3d592160212f78ff77df7d4: Status 404 returned error can't find the container with id 0d9ae068ac8297dbbf5386bcbecd7a6c50a63e03d3d592160212f78ff77df7d4 Sep 16 12:48:15.549618 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.549506 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9zltf" Sep 16 12:48:15.552415 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:15.552383 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37248adf_9ac8_4fd6_a693_daf752b60555.slice/crio-9819d3fbbeb22301cbece3c7a459e169cd3471babc02ecd4bbabcf7e9b912c91 WatchSource:0}: Error finding container 9819d3fbbeb22301cbece3c7a459e169cd3471babc02ecd4bbabcf7e9b912c91: Status 404 returned error can't find the container with id 9819d3fbbeb22301cbece3c7a459e169cd3471babc02ecd4bbabcf7e9b912c91 Sep 16 12:48:15.557647 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:15.555871 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b18e641_43cd_4ad1_b68e_63d3427fc9fb.slice/crio-484b244889667ba6ae1554ac076166cdfa14b741636b6cfa3acc6d3415693439 WatchSource:0}: Error finding container 484b244889667ba6ae1554ac076166cdfa14b741636b6cfa3acc6d3415693439: Status 404 returned error can't find the container with id 484b244889667ba6ae1554ac076166cdfa14b741636b6cfa3acc6d3415693439 Sep 16 12:48:15.557647 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.557310 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2259d" Sep 16 12:48:15.562542 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.562523 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bslcs" Sep 16 12:48:15.562721 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:15.562699 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda716d8_6057_42d2_8a6f_ee5cddd8592a.slice/crio-cf4d974228b9285e8d2b2ae4f745cf5f53e3a5a0fa09d6d189a991aee94541d3 WatchSource:0}: Error finding container cf4d974228b9285e8d2b2ae4f745cf5f53e3a5a0fa09d6d189a991aee94541d3: Status 404 returned error can't find the container with id cf4d974228b9285e8d2b2ae4f745cf5f53e3a5a0fa09d6d189a991aee94541d3 Sep 16 12:48:15.575824 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:15.575799 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f90e3ec_14f4_443f_b93a_1a1d05265a71.slice/crio-b3aec3af239257eda11939120681bae005a37172b304386578fbcbfff72662e2 WatchSource:0}: Error finding container b3aec3af239257eda11939120681bae005a37172b304386578fbcbfff72662e2: Status 404 returned error can't find the container with id b3aec3af239257eda11939120681bae005a37172b304386578fbcbfff72662e2 Sep 16 12:48:15.819847 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.819746 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:15.820049 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.819934 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:15.820049 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.820024 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs podName:418a1352-4779-435a-880c-627e3480b111 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:16.82000129 +0000 UTC m=+4.012649875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs") pod "network-metrics-daemon-jjdcd" (UID: "418a1352-4779-435a-880c-627e3480b111") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:15.919199 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.919163 2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 12:48:15.921167 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.921095 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtnp\" (UniqueName: \"kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp\") pod \"network-check-target-btfcd\" (UID: \"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04\") " pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:15.921167 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:15.921146 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:15.921424 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.921331 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:15.921424 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.921405 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret podName:00648523-fd55-498f-b8c6-704f6c9d7e47 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:16.921384799 +0000 UTC m=+4.114033373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret") pod "global-pull-secret-syncer-hrjnz" (UID: "00648523-fd55-498f-b8c6-704f6c9d7e47") : object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:15.921798 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.921774 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 12:48:15.921929 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.921805 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 12:48:15.921929 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.921819 2584 projected.go:196] Error preparing data for projected volume kube-api-access-2dtnp for pod openshift-network-diagnostics/network-check-target-btfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:15.921929 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:15.921884 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp podName:aacaaf0a-8a6f-4ced-9c88-1fab1496cb04 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:16.921865648 +0000 UTC m=+4.114514240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2dtnp" (UniqueName: "kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp") pod "network-check-target-btfcd" (UID: "aacaaf0a-8a6f-4ced-9c88-1fab1496cb04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:16.253886 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.253804 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:43:15 +0000 UTC" deadline="2028-05-24 11:21:46.06297295 +0000 UTC" Sep 16 12:48:16.253886 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.253865 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14782h33m29.809113177s" Sep 16 12:48:16.467787 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.467636 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:16.467964 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:16.467784 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:16.477609 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.477502 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" event={"ID":"3b18e641-43cd-4ad1-b68e-63d3427fc9fb","Type":"ContainerStarted","Data":"484b244889667ba6ae1554ac076166cdfa14b741636b6cfa3acc6d3415693439"} Sep 16 12:48:16.479495 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.479145 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g7gz4" event={"ID":"b868f668-a89e-43dd-bb29-18c856372b90","Type":"ContainerStarted","Data":"0d9ae068ac8297dbbf5386bcbecd7a6c50a63e03d3d592160212f78ff77df7d4"} Sep 16 12:48:16.481028 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.481000 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gbhf2" event={"ID":"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc","Type":"ContainerStarted","Data":"3d7c3d361e914cf7bb00f43b86a2232bf896098b9496b4581485bc9b7e14f92d"} Sep 16 12:48:16.482688 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.482662 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bslcs" event={"ID":"be40f43f-3d2a-4499-8e88-b78cd35bc9ed","Type":"ContainerStarted","Data":"cac548903c09965163d971ca26387cb18b458d9d5307e3c307d48ab26e2f75cf"} Sep 16 12:48:16.484281 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.484252 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2259d" event={"ID":"4f90e3ec-14f4-443f-b93a-1a1d05265a71","Type":"ContainerStarted","Data":"b3aec3af239257eda11939120681bae005a37172b304386578fbcbfff72662e2"} Sep 16 12:48:16.485706 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.485681 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9zltf" event={"ID":"dda716d8-6057-42d2-8a6f-ee5cddd8592a","Type":"ContainerStarted","Data":"cf4d974228b9285e8d2b2ae4f745cf5f53e3a5a0fa09d6d189a991aee94541d3"} Sep 16 12:48:16.487376 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.487347 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fgbqm" event={"ID":"37248adf-9ac8-4fd6-a693-daf752b60555","Type":"ContainerStarted","Data":"9819d3fbbeb22301cbece3c7a459e169cd3471babc02ecd4bbabcf7e9b912c91"} Sep 16 12:48:16.489209 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.489151 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c64wg" event={"ID":"9711ead1-9753-4be1-b0f5-78cd0ac6f89f","Type":"ContainerStarted","Data":"514cca67befc9d08e1133c2393ca83f0982871b676847947b07cb7cf32b58b13"} Sep 16 12:48:16.490943 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.490920 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" event={"ID":"10c6c3a7-6ac2-45ea-9726-28260b38b6d8","Type":"ContainerStarted","Data":"d75af67e95333bd073ba587847c96cb7e599e7df9f4d98e460a0dfda8236a19a"} Sep 16 12:48:16.829445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.828855 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:16.829445 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:16.829009 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:16.829445 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:16.829079 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs podName:418a1352-4779-435a-880c-627e3480b111 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:18.829058594 +0000 UTC m=+6.021707169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs") pod "network-metrics-daemon-jjdcd" (UID: "418a1352-4779-435a-880c-627e3480b111") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:16.929293 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.929252 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtnp\" (UniqueName: \"kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp\") pod \"network-check-target-btfcd\" (UID: \"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04\") " pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:16.929464 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:16.929316 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:16.929542 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:16.929487 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:16.929542 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:16.929503 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 12:48:16.929542 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:16.929520 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 12:48:16.929542 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:16.929533 2584 projected.go:196] Error preparing data for projected volume kube-api-access-2dtnp for pod openshift-network-diagnostics/network-check-target-btfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:16.929766 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:16.929560 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret podName:00648523-fd55-498f-b8c6-704f6c9d7e47 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:18.929539589 +0000 UTC m=+6.122188185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret") pod "global-pull-secret-syncer-hrjnz" (UID: "00648523-fd55-498f-b8c6-704f6c9d7e47") : object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:16.929766 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:16.929599 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp podName:aacaaf0a-8a6f-4ced-9c88-1fab1496cb04 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:18.929588966 +0000 UTC m=+6.122237543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2dtnp" (UniqueName: "kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp") pod "network-check-target-btfcd" (UID: "aacaaf0a-8a6f-4ced-9c88-1fab1496cb04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:17.254288 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:17.254236 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:43:15 +0000 UTC" deadline="2028-05-26 05:30:36.066334036 +0000 UTC" Sep 16 12:48:17.254288 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:17.254289 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14824h42m18.812050788s" Sep 16 12:48:17.467924 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:17.467889 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:17.468158 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:17.467896 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:17.468158 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:17.468038 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:17.468158 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:17.468126 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:18.468108 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:18.467848 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:18.468596 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:18.468254 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:18.849310 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:18.848628 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:18.849310 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:18.848843 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:18.849310 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:18.848910 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs podName:418a1352-4779-435a-880c-627e3480b111 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:22.848890441 +0000 UTC m=+10.041539037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs") pod "network-metrics-daemon-jjdcd" (UID: "418a1352-4779-435a-880c-627e3480b111") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:18.950345 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:18.950310 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtnp\" (UniqueName: \"kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp\") pod \"network-check-target-btfcd\" (UID: \"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04\") " pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:18.950518 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:18.950358 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:18.950518 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:18.950479 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:18.950656 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:18.950539 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret podName:00648523-fd55-498f-b8c6-704f6c9d7e47 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:22.950522017 +0000 UTC m=+10.143170590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret") pod "global-pull-secret-syncer-hrjnz" (UID: "00648523-fd55-498f-b8c6-704f6c9d7e47") : object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:18.951136 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:18.950938 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 12:48:18.951136 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:18.950964 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 12:48:18.951136 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:18.950987 2584 projected.go:196] Error preparing data for projected volume kube-api-access-2dtnp for pod openshift-network-diagnostics/network-check-target-btfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:18.951136 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:18.951026 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp podName:aacaaf0a-8a6f-4ced-9c88-1fab1496cb04 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:22.951013236 +0000 UTC m=+10.143661817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2dtnp" (UniqueName: "kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp") pod "network-check-target-btfcd" (UID: "aacaaf0a-8a6f-4ced-9c88-1fab1496cb04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:19.468292 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:19.468247 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:19.468292 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:19.468292 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:19.468783 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:19.468381 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:19.468843 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:19.468804 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:20.467466 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:20.467431 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:20.467657 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:20.467550 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:21.467589 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:21.467558 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:21.468079 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:21.467689 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:21.468079 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:21.468032 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:21.468268 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:21.468144 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:22.467518 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:22.467411 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:22.467681 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:22.467538 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:22.884646 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:22.884032 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:22.884646 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:22.884185 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:22.884646 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:22.884256 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs podName:418a1352-4779-435a-880c-627e3480b111 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:30.884235664 +0000 UTC m=+18.076884236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs") pod "network-metrics-daemon-jjdcd" (UID: "418a1352-4779-435a-880c-627e3480b111") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:22.985061 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:22.985016 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtnp\" (UniqueName: \"kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp\") pod \"network-check-target-btfcd\" (UID: \"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04\") " pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:22.985265 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:22.985072 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:22.985265 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:22.985206 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:22.985387 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:22.985269 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret podName:00648523-fd55-498f-b8c6-704f6c9d7e47 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:30.985251656 +0000 UTC m=+18.177900228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret") pod "global-pull-secret-syncer-hrjnz" (UID: "00648523-fd55-498f-b8c6-704f6c9d7e47") : object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:22.985387 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:22.985340 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 12:48:22.985387 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:22.985368 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 12:48:22.985387 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:22.985382 2584 projected.go:196] Error preparing data for projected volume kube-api-access-2dtnp for pod openshift-network-diagnostics/network-check-target-btfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:22.985597 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:22.985423 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp podName:aacaaf0a-8a6f-4ced-9c88-1fab1496cb04 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:30.985411714 +0000 UTC m=+18.178060289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2dtnp" (UniqueName: "kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp") pod "network-check-target-btfcd" (UID: "aacaaf0a-8a6f-4ced-9c88-1fab1496cb04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:23.468876 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:23.468841 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:23.469347 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:23.468971 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:23.469347 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:23.469337 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:23.469500 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:23.469442 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:24.467484 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:24.467451 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:24.467707 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:24.467580 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:25.467636 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:25.467606 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:25.468065 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:25.467746 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:25.468065 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:25.467799 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:25.468065 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:25.467917 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:26.467119 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:26.467045 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:26.467263 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:26.467150 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:27.467133 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:27.467099 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:27.467582 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:27.467278 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:27.467582 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:27.467343 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:27.467582 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:27.467459 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:28.467370 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:28.467331 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:28.467747 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:28.467468 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:29.467693 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:29.467657 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:29.468088 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:29.467796 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:29.468088 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:29.467852 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:29.468088 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:29.467984 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:30.467102 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:30.467073 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:30.467347 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:30.467206 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:30.939352 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:30.939319 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:30.939769 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:30.939440 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:30.939769 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:30.939528 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs podName:418a1352-4779-435a-880c-627e3480b111 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:46.939508229 +0000 UTC m=+34.132156796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs") pod "network-metrics-daemon-jjdcd" (UID: "418a1352-4779-435a-880c-627e3480b111") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:31.040441 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:31.040406 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtnp\" (UniqueName: \"kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp\") pod \"network-check-target-btfcd\" (UID: \"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04\") " pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:31.040684 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:31.040458 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:31.040684 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:31.040572 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:31.040684 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:31.040632 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret podName:00648523-fd55-498f-b8c6-704f6c9d7e47 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.040618634 +0000 UTC m=+34.233267201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret") pod "global-pull-secret-syncer-hrjnz" (UID: "00648523-fd55-498f-b8c6-704f6c9d7e47") : object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:31.040684 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:31.040574 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 12:48:31.040684 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:31.040685 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 12:48:31.040903 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:31.040697 2584 projected.go:196] Error preparing data for projected volume kube-api-access-2dtnp for pod openshift-network-diagnostics/network-check-target-btfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:31.040903 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:31.040740 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp podName:aacaaf0a-8a6f-4ced-9c88-1fab1496cb04 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.040728619 +0000 UTC m=+34.233377202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2dtnp" (UniqueName: "kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp") pod "network-check-target-btfcd" (UID: "aacaaf0a-8a6f-4ced-9c88-1fab1496cb04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:31.467364 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:31.467330 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:31.467555 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:31.467347 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:31.467555 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:31.467491 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:31.467667 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:31.467597 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:32.467483 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:32.467450 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:32.468031 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:32.467549 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:33.468208 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:33.467886 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:33.468208 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:33.468064 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:33.468208 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:33.468152 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:33.468957 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:33.467996 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:33.529670 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:33.529637 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" event={"ID":"10c6c3a7-6ac2-45ea-9726-28260b38b6d8","Type":"ContainerStarted","Data":"1da81c3d691668ecab9c8d43012ab2222ec5e4a1f9ba287c2bb7ad3b970b2ec3"} Sep 16 12:48:33.531284 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:33.531257 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" event={"ID":"8b76f1b2a16574ce38927f7f5da97679","Type":"ContainerStarted","Data":"90436a7bfcbe3be932241ff68a7b9d190b33ce3ef26c0c5a1ca87ad0f8fcaf8d"} Sep 16 12:48:33.532967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:33.532940 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9zltf" event={"ID":"dda716d8-6057-42d2-8a6f-ee5cddd8592a","Type":"ContainerStarted","Data":"8ec1029ef63f968156601e7cfd223f4fff38770a8eede9144f5c3ffcfa8d9a89"} Sep 16 12:48:33.555217 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:33.555154 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-139.ec2.internal" podStartSLOduration=19.555142707999998 podStartE2EDuration="19.555142708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:33.554855154 +0000 UTC m=+20.747503743" watchObservedRunningTime="2026-09-16 12:48:33.555142708 +0000 UTC m=+20.747791296" Sep 16 12:48:34.467482 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.467142 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:34.467649 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:34.467558 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:34.536283 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.536251 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c64wg" event={"ID":"9711ead1-9753-4be1-b0f5-78cd0ac6f89f","Type":"ContainerStarted","Data":"6cc47a7f71410581395abb4a673f3f6c657ccb0aea07d9bb22ad1214ba611658"} Sep 16 12:48:34.538907 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.538887 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 12:48:34.539202 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.539185 2584 generic.go:356] "Generic (PLEG): container finished" podID="10c6c3a7-6ac2-45ea-9726-28260b38b6d8" containerID="fea8fc9b2eb7ed81d9447bd32529385ef720b227cc359a57c4c8108a37fa3b35" exitCode=1 Sep 16 12:48:34.539266 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.539249 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" event={"ID":"10c6c3a7-6ac2-45ea-9726-28260b38b6d8","Type":"ContainerStarted","Data":"417643a2e8a9137698cafbb18a883063dcd0375ca9189b25cebde1988a5de7ea"} Sep 16 12:48:34.539333 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.539319 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" event={"ID":"10c6c3a7-6ac2-45ea-9726-28260b38b6d8","Type":"ContainerStarted","Data":"cfe0f095ca85f1c2384835f5d5e7e7f455f0e656ee4c0a83d76680660937929d"} Sep 16 12:48:34.539376 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.539365 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" event={"ID":"10c6c3a7-6ac2-45ea-9726-28260b38b6d8","Type":"ContainerStarted","Data":"98b370367078d4169a29dfccd63500546f794b0afb9e482ce0115b7b6016d469"} Sep 16 12:48:34.539419 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.539404 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" event={"ID":"10c6c3a7-6ac2-45ea-9726-28260b38b6d8","Type":"ContainerStarted","Data":"97678b821f0fd6422e25dba9a4fafccbe7a57667185bed01068a930ae9eedd49"} Sep 16 12:48:34.539476 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.539439 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" event={"ID":"10c6c3a7-6ac2-45ea-9726-28260b38b6d8","Type":"ContainerDied","Data":"fea8fc9b2eb7ed81d9447bd32529385ef720b227cc359a57c4c8108a37fa3b35"} Sep 16 12:48:34.540725 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.540704 2584 generic.go:356] "Generic (PLEG): container finished" podID="628f3dc115ebc1acc6ed7da71915e638" containerID="f07449cae90cb2d7f0e68546626cb176b8712ce242a9834a1e22b8d9d74203ee" exitCode=0 Sep 16 12:48:34.540810 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.540789 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" event={"ID":"628f3dc115ebc1acc6ed7da71915e638","Type":"ContainerDied","Data":"f07449cae90cb2d7f0e68546626cb176b8712ce242a9834a1e22b8d9d74203ee"} Sep 16 12:48:34.542142 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.542126 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" event={"ID":"3b18e641-43cd-4ad1-b68e-63d3427fc9fb","Type":"ContainerStarted","Data":"2fb9890efd0c76cf7cdabeca9cee580c11dac62d5cd73b917f4807104cadeea4"} Sep 16 12:48:34.543640 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.543623 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g7gz4" event={"ID":"b868f668-a89e-43dd-bb29-18c856372b90","Type":"ContainerStarted","Data":"a51f6385c84602571c6363f8bc9cd5b84246e2f3e24c6def2e64a3dfe888663a"} Sep 16 12:48:34.545143 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.545068 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gbhf2" event={"ID":"b6555a32-a5f6-4d65-82c7-6aaabaff1fbc","Type":"ContainerStarted","Data":"6a30afafe37b347739f8c0415bc0ac7d62f21241d46df21245c43a9e98da0d48"} Sep 16 12:48:34.546435 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.546414 2584 generic.go:356] "Generic (PLEG): container finished" podID="be40f43f-3d2a-4499-8e88-b78cd35bc9ed" containerID="2f23b3115c5b69c589d49599559daea0840fcc89a79327a019912d0b4fa7c8c7" exitCode=0 Sep 16 12:48:34.546529 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.546478 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bslcs" event={"ID":"be40f43f-3d2a-4499-8e88-b78cd35bc9ed","Type":"ContainerDied","Data":"2f23b3115c5b69c589d49599559daea0840fcc89a79327a019912d0b4fa7c8c7"} Sep 16 12:48:34.548083 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.548060 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fgbqm" event={"ID":"37248adf-9ac8-4fd6-a693-daf752b60555","Type":"ContainerStarted","Data":"e958a76352e57bca22ebb0ce1498df20f24dedd26ee4b1d0e6f4285ca19363a7"} Sep 16 12:48:34.554117 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.554081 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9zltf" podStartSLOduration=3.846677891 podStartE2EDuration="21.554068547s" totalImagesPullingTime="17.707390656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:33.578509323 +0000 UTC m=+20.771157914" watchObservedRunningTime="2026-09-16 12:48:34.554068547 +0000 UTC m=+21.746717207" Sep 16 12:48:34.554218 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.554187 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c64wg" podStartSLOduration=14.471072202 podStartE2EDuration="21.554180224s" totalImagesPullingTime="7.083108022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:34.553653728 +0000 UTC m=+21.746302330" watchObservedRunningTime="2026-09-16 12:48:34.554180224 +0000 UTC m=+21.746828811" Sep 16 12:48:34.577440 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.577339 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g7gz4" podStartSLOduration=3.429048132 podStartE2EDuration="21.577318152s" totalImagesPullingTime="18.14827002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:34.57675269 +0000 UTC m=+21.769401302" watchObservedRunningTime="2026-09-16 12:48:34.577318152 +0000 UTC m=+21.769966745" Sep 16 12:48:34.673922 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.673874 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fgbqm" podStartSLOduration=3.955392599 podStartE2EDuration="21.673855179s" totalImagesPullingTime="17.71846258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:34.673443145 +0000 UTC m=+21.866091745" watchObservedRunningTime="2026-09-16 12:48:34.673855179 +0000 UTC m=+21.866503767" Sep 16 12:48:34.722502 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.722455 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gbhf2" podStartSLOduration=14.650101413 podStartE2EDuration="21.722435524s" totalImagesPullingTime="7.072334111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:34.722034732 +0000 UTC m=+21.914683339" watchObservedRunningTime="2026-09-16 12:48:34.722435524 +0000 UTC m=+21.915084173" Sep 16 12:48:34.814177 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.814125 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:34.900081 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.900003 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:34.996737 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:34.996685 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:35.030964 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.030940 2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 12:48:35.102130 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.102104 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:35.260338 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.260234 2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:48:35.030961852Z","UUID":"1c8132b4-45b0-43cc-8f7c-7a7a161ed4c0","Handler":null,"Name":"","Endpoint":""} Sep 16 12:48:35.261863 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.261838 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 12:48:35.261973 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.261873 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 12:48:35.467967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.467930 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:35.467967 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.467957 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:35.468221 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:35.468049 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:35.468283 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:35.468221 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:35.553202 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.553104 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" event={"ID":"628f3dc115ebc1acc6ed7da71915e638","Type":"ContainerStarted","Data":"0748942ce47b19f86de025aac97f50d0f1ee1e253ef6571ee53f7c4431bb9bce"} Sep 16 12:48:35.555292 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.555264 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" event={"ID":"3b18e641-43cd-4ad1-b68e-63d3427fc9fb","Type":"ContainerStarted","Data":"30737a6ac4af4756c2080ee4cc3dd0627c4f4163b7987cdb4e38759c8c072051"} Sep 16 12:48:35.556958 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.556919 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2259d" event={"ID":"4f90e3ec-14f4-443f-b93a-1a1d05265a71","Type":"ContainerStarted","Data":"9a5bbf1394b37fb41de0c0dd7e2b5e66a5fbbfc5bc417600a1ae4f9d6248c869"} Sep 16 12:48:35.577544 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.577497 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-139.ec2.internal" podStartSLOduration=21.577481345 podStartE2EDuration="21.577481345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:35.576881011 +0000 UTC m=+22.769529623" watchObservedRunningTime="2026-09-16 12:48:35.577481345 +0000 UTC m=+22.770129935" Sep 16 12:48:35.796894 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.796858 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:35.885457 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:35.885366 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:36.077531 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:36.077499 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:36.178450 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:36.178424 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:36.467134 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:36.467066 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:36.467301 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:36.467187 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:36.561534 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:36.561494 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" event={"ID":"3b18e641-43cd-4ad1-b68e-63d3427fc9fb","Type":"ContainerStarted","Data":"8e544443f3ebfb466c14a0b486e951305ad9d6ffa0917b99c1b9618a26f2a3d6"} Sep 16 12:48:36.565227 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:36.565194 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 12:48:36.565660 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:36.565629 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" event={"ID":"10c6c3a7-6ac2-45ea-9726-28260b38b6d8","Type":"ContainerStarted","Data":"f5e5d3fe347e4f6860722229b245f2bfb2f2beb30b16cf07d91caa8e74a5de99"} Sep 16 12:48:36.584171 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:36.584127 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2259d" podStartSLOduration=6.113563779 podStartE2EDuration="23.584108398s" totalImagesPullingTime="17.470544619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:35.598417695 +0000 UTC m=+22.791066286" watchObservedRunningTime="2026-09-16 12:48:36.584108398 +0000 UTC m=+23.776756992" Sep 16 12:48:36.584349 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:36.584210 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-46szn" podStartSLOduration=3.850711134 podStartE2EDuration="23.584203752s" totalImagesPullingTime="19.733492618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:36.583807018 +0000 UTC m=+23.776455625" watchObservedRunningTime="2026-09-16 12:48:36.584203752 +0000 UTC m=+23.776852341" Sep 16 12:48:36.745851 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:36.745816 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:36.845632 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:36.845589 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:37.467153 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:37.466956 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:37.467341 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:37.466999 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:37.467341 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:37.467257 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:37.467455 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:37.467333 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:37.656430 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:37.656399 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:37.657192 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:37.657174 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:38.467417 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:38.467385 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:38.467621 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:38.467497 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:39.467069 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:39.467036 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:39.467556 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:39.467139 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:39.467556 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:39.467222 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:39.467556 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:39.467320 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:39.573477 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:39.573300 2584 generic.go:356] "Generic (PLEG): container finished" podID="be40f43f-3d2a-4499-8e88-b78cd35bc9ed" containerID="2792020a7e3fa7343eb709379bdb76525b9d583889c3bde1b0149408851eef2d" exitCode=0 Sep 16 12:48:39.573639 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:39.573385 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bslcs" event={"ID":"be40f43f-3d2a-4499-8e88-b78cd35bc9ed","Type":"ContainerDied","Data":"2792020a7e3fa7343eb709379bdb76525b9d583889c3bde1b0149408851eef2d"} Sep 16 12:48:39.577524 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:39.577504 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 12:48:39.577949 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:39.577928 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" event={"ID":"10c6c3a7-6ac2-45ea-9726-28260b38b6d8","Type":"ContainerStarted","Data":"ddb167230f43b94f7645b5f199d1b6ccde83e5bb0a7bf4cbff15e5c462841a36"} Sep 16 12:48:39.578185 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:39.578154 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:39.578395 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:39.578378 2584 scope.go:117] "RemoveContainer" containerID="fea8fc9b2eb7ed81d9447bd32529385ef720b227cc359a57c4c8108a37fa3b35" Sep 16 12:48:39.593588 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:39.593570 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:40.467247 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.467214 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:40.467791 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:40.467308 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:40.503859 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.503827 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrjnz"] Sep 16 12:48:40.503998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.503923 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:40.504046 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:40.503994 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:40.508911 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.508844 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-btfcd"] Sep 16 12:48:40.509626 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.509607 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jjdcd"] Sep 16 12:48:40.509719 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.509695 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:40.509782 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:40.509766 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:40.585347 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.585315 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 12:48:40.585804 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.585735 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:40.585804 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.585777 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" event={"ID":"10c6c3a7-6ac2-45ea-9726-28260b38b6d8","Type":"ContainerStarted","Data":"2b3c9e53cdd2f85bcc496b2d08c9de95b5c821940503a5cff0e02f0abbedca44"} Sep 16 12:48:40.586003 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:40.585838 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:40.586003 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.585878 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:40.586003 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.585923 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:40.602911 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.602891 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:48:40.620149 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:40.620096 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" podStartSLOduration=9.808679936 podStartE2EDuration="27.620078475s" totalImagesPullingTime="17.811398539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:40.618147204 +0000 UTC m=+27.810795834" watchObservedRunningTime="2026-09-16 12:48:40.620078475 +0000 UTC m=+27.812727063" Sep 16 12:48:41.589424 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:41.589394 2584 generic.go:356] "Generic (PLEG): container finished" podID="be40f43f-3d2a-4499-8e88-b78cd35bc9ed" containerID="fac6975da8e74b158068ea40e57f7726a081dc6794b1f8a99169d40a545c0717" exitCode=0 Sep 16 12:48:41.589923 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:41.589478 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bslcs" event={"ID":"be40f43f-3d2a-4499-8e88-b78cd35bc9ed","Type":"ContainerDied","Data":"fac6975da8e74b158068ea40e57f7726a081dc6794b1f8a99169d40a545c0717"} Sep 16 12:48:42.467438 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:42.467405 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:42.467562 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:42.467443 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:42.467562 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:42.467445 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:42.467562 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:42.467530 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:42.467724 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:42.467589 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:42.467724 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:42.467680 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:42.593993 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:42.593966 2584 generic.go:356] "Generic (PLEG): container finished" podID="be40f43f-3d2a-4499-8e88-b78cd35bc9ed" containerID="5f9a82439e6357bac6ed4d09166b1c713d3173df77f39f6124a3e161f03aafb7" exitCode=0 Sep 16 12:48:42.594327 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:42.594055 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bslcs" event={"ID":"be40f43f-3d2a-4499-8e88-b78cd35bc9ed","Type":"ContainerDied","Data":"5f9a82439e6357bac6ed4d09166b1c713d3173df77f39f6124a3e161f03aafb7"} Sep 16 12:48:44.467912 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:44.467695 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:44.468390 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:44.467695 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:44.468390 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:44.468012 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:44.468390 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:44.467695 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:44.468390 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:44.468103 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:44.468390 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:44.468189 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:45.135037 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:45.134082 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:45.135037 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:45.134238 2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 16 12:48:45.135037 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:45.134773 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fgbqm" Sep 16 12:48:45.233219 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:45.233146 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:46.467073 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.467038 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:46.467510 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.467154 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:46.467510 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:46.467154 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jjdcd" podUID="418a1352-4779-435a-880c-627e3480b111" Sep 16 12:48:46.467510 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.467178 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:46.467510 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:46.467293 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrjnz" podUID="00648523-fd55-498f-b8c6-704f6c9d7e47" Sep 16 12:48:46.467510 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:46.467319 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btfcd" podUID="aacaaf0a-8a6f-4ced-9c88-1fab1496cb04" Sep 16 12:48:46.576774 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.576705 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-139.ec2.internal" event="NodeReady" Sep 16 12:48:46.576898 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.576841 2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 12:48:46.635313 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.635287 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-st6jz"] Sep 16 12:48:46.648986 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.648968 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vvb26"] Sep 16 12:48:46.649100 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.649083 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-st6jz" Sep 16 12:48:46.652526 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.652509 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 12:48:46.652634 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.652617 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 12:48:46.652852 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.652840 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-54qld\"" Sep 16 12:48:46.653211 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.653192 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 12:48:46.669108 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.669087 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.669198 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.669108 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-st6jz"] Sep 16 12:48:46.672926 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.672907 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vvb26"] Sep 16 12:48:46.674939 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.674920 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 12:48:46.675009 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.674935 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sl6fp\"" Sep 16 12:48:46.675045 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.675019 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 12:48:46.750386 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.750353 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.750386 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.750386 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42ae37fd-a147-4072-b59d-ebbda3c45206-config-volume\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.750565 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.750430 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49a4be54-687a-4031-9487-72e8d73bdc3d-cert\") pod \"ingress-canary-st6jz\" (UID: \"49a4be54-687a-4031-9487-72e8d73bdc3d\") " pod="openshift-ingress-canary/ingress-canary-st6jz" Sep 16 12:48:46.750565 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.750449 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ae37fd-a147-4072-b59d-ebbda3c45206-tmp-dir\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.750565 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.750471 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqh4k\" (UniqueName: \"kubernetes.io/projected/49a4be54-687a-4031-9487-72e8d73bdc3d-kube-api-access-cqh4k\") pod \"ingress-canary-st6jz\" (UID: \"49a4be54-687a-4031-9487-72e8d73bdc3d\") " pod="openshift-ingress-canary/ingress-canary-st6jz" Sep 16 12:48:46.750565 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.750486 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9wk\" (UniqueName: \"kubernetes.io/projected/42ae37fd-a147-4072-b59d-ebbda3c45206-kube-api-access-9r9wk\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.837396 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.837326 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:46.852608 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.851325 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.852608 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.851393 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42ae37fd-a147-4072-b59d-ebbda3c45206-config-volume\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.852608 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.851499 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49a4be54-687a-4031-9487-72e8d73bdc3d-cert\") pod \"ingress-canary-st6jz\" (UID: \"49a4be54-687a-4031-9487-72e8d73bdc3d\") " pod="openshift-ingress-canary/ingress-canary-st6jz" Sep 16 12:48:46.852608 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.851548 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ae37fd-a147-4072-b59d-ebbda3c45206-tmp-dir\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.852608 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:46.851568 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 12:48:46.852608 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.851602 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqh4k\" (UniqueName: \"kubernetes.io/projected/49a4be54-687a-4031-9487-72e8d73bdc3d-kube-api-access-cqh4k\") pod \"ingress-canary-st6jz\" (UID: \"49a4be54-687a-4031-9487-72e8d73bdc3d\") " pod="openshift-ingress-canary/ingress-canary-st6jz" Sep 16 12:48:46.852608 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:46.851640 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls podName:42ae37fd-a147-4072-b59d-ebbda3c45206 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.351612123 +0000 UTC m=+34.544260703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls") pod "dns-default-vvb26" (UID: "42ae37fd-a147-4072-b59d-ebbda3c45206") : secret "dns-default-metrics-tls" not found Sep 16 12:48:46.852608 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.851677 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9wk\" (UniqueName: \"kubernetes.io/projected/42ae37fd-a147-4072-b59d-ebbda3c45206-kube-api-access-9r9wk\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.852608 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.852296 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42ae37fd-a147-4072-b59d-ebbda3c45206-config-volume\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.852608 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.852553 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ae37fd-a147-4072-b59d-ebbda3c45206-tmp-dir\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.857041 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.857022 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49a4be54-687a-4031-9487-72e8d73bdc3d-cert\") pod \"ingress-canary-st6jz\" (UID: \"49a4be54-687a-4031-9487-72e8d73bdc3d\") " pod="openshift-ingress-canary/ingress-canary-st6jz" Sep 16 12:48:46.864253 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.864232 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqh4k\" (UniqueName: \"kubernetes.io/projected/49a4be54-687a-4031-9487-72e8d73bdc3d-kube-api-access-cqh4k\") pod \"ingress-canary-st6jz\" (UID: \"49a4be54-687a-4031-9487-72e8d73bdc3d\") " pod="openshift-ingress-canary/ingress-canary-st6jz" Sep 16 12:48:46.864392 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.864375 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9wk\" (UniqueName: \"kubernetes.io/projected/42ae37fd-a147-4072-b59d-ebbda3c45206-kube-api-access-9r9wk\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:46.941080 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.941048 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:46.952593 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.952571 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:46.952711 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:46.952696 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:46.952777 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:46.952768 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs podName:418a1352-4779-435a-880c-627e3480b111 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.952750507 +0000 UTC m=+66.145399085 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs") pod "network-metrics-daemon-jjdcd" (UID: "418a1352-4779-435a-880c-627e3480b111") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 12:48:46.958578 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:46.958560 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-st6jz" Sep 16 12:48:47.053179 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:47.053127 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtnp\" (UniqueName: \"kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp\") pod \"network-check-target-btfcd\" (UID: \"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04\") " pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:47.053317 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:47.053195 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:47.053317 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:47.053310 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 12:48:47.053433 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:47.053331 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 12:48:47.053433 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:47.053353 2584 projected.go:196] Error preparing data for projected volume kube-api-access-2dtnp for pod openshift-network-diagnostics/network-check-target-btfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:47.053433 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:47.053415 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp podName:aacaaf0a-8a6f-4ced-9c88-1fab1496cb04 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:19.053395016 +0000 UTC m=+66.246043609 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2dtnp" (UniqueName: "kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp") pod "network-check-target-btfcd" (UID: "aacaaf0a-8a6f-4ced-9c88-1fab1496cb04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 12:48:47.053908 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:47.053885 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:47.054072 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:47.053962 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret podName:00648523-fd55-498f-b8c6-704f6c9d7e47 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:19.053945704 +0000 UTC m=+66.246594285 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret") pod "global-pull-secret-syncer-hrjnz" (UID: "00648523-fd55-498f-b8c6-704f6c9d7e47") : object "kube-system"/"original-pull-secret" not registered Sep 16 12:48:47.290359 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:47.290205 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-st6jz"] Sep 16 12:48:47.294365 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:48:47.294337 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a4be54_687a_4031_9487_72e8d73bdc3d.slice/crio-27070270f4a84f9ad74f17ea30c375026a861b5a2bbc5ee9f3e2815f8b4d99b3 WatchSource:0}: Error finding container 27070270f4a84f9ad74f17ea30c375026a861b5a2bbc5ee9f3e2815f8b4d99b3: Status 404 returned error can't find the container with id 27070270f4a84f9ad74f17ea30c375026a861b5a2bbc5ee9f3e2815f8b4d99b3 Sep 16 12:48:47.355525 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:47.355473 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:47.355598 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:47.355576 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 12:48:47.355673 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:47.355632 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls podName:42ae37fd-a147-4072-b59d-ebbda3c45206 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:48.355613718 +0000 UTC m=+35.548262288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls") pod "dns-default-vvb26" (UID: "42ae37fd-a147-4072-b59d-ebbda3c45206") : secret "dns-default-metrics-tls" not found Sep 16 12:48:47.606063 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:47.605973 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-st6jz" event={"ID":"49a4be54-687a-4031-9487-72e8d73bdc3d","Type":"ContainerStarted","Data":"27070270f4a84f9ad74f17ea30c375026a861b5a2bbc5ee9f3e2815f8b4d99b3"} Sep 16 12:48:47.608647 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:47.608622 2584 generic.go:356] "Generic (PLEG): container finished" podID="be40f43f-3d2a-4499-8e88-b78cd35bc9ed" containerID="4835aa7f98d787f736e5e1ea6a4d53e197447466729437740c29d2fc1965dfdf" exitCode=0 Sep 16 12:48:47.608777 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:47.608676 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bslcs" event={"ID":"be40f43f-3d2a-4499-8e88-b78cd35bc9ed","Type":"ContainerDied","Data":"4835aa7f98d787f736e5e1ea6a4d53e197447466729437740c29d2fc1965dfdf"} Sep 16 12:48:48.363360 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.363328 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:48.363510 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:48.363491 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 12:48:48.363572 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:48.363560 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls podName:42ae37fd-a147-4072-b59d-ebbda3c45206 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:50.363540247 +0000 UTC m=+37.556188834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls") pod "dns-default-vvb26" (UID: "42ae37fd-a147-4072-b59d-ebbda3c45206") : secret "dns-default-metrics-tls" not found Sep 16 12:48:48.467796 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.467764 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:48:48.468012 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.467998 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:48:48.468098 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.468079 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:48:48.472969 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.472947 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 12:48:48.473089 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.472988 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-27x5j\"" Sep 16 12:48:48.473089 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.473016 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 12:48:48.473231 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.472952 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 12:48:48.473231 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.472947 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-djtjx\"" Sep 16 12:48:48.473369 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.473354 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 12:48:48.614247 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.614184 2584 generic.go:356] "Generic (PLEG): container finished" podID="be40f43f-3d2a-4499-8e88-b78cd35bc9ed" containerID="a9c22bb6eb03b66f854dc12255a77ebdc2c96f8ce58f28553ac1ddce9d210a44" exitCode=0 Sep 16 12:48:48.614247 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:48.614239 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bslcs" event={"ID":"be40f43f-3d2a-4499-8e88-b78cd35bc9ed","Type":"ContainerDied","Data":"a9c22bb6eb03b66f854dc12255a77ebdc2c96f8ce58f28553ac1ddce9d210a44"} Sep 16 12:48:49.618233 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:49.618174 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-st6jz" event={"ID":"49a4be54-687a-4031-9487-72e8d73bdc3d","Type":"ContainerStarted","Data":"bd4796b6419c285d4e73f3e2cc0200d777cbe5a2782f1e760adc4d9b13e3f751"} Sep 16 12:48:49.621672 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:49.621646 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bslcs" event={"ID":"be40f43f-3d2a-4499-8e88-b78cd35bc9ed","Type":"ContainerStarted","Data":"45dec66579d0f44f57d9b9be787a6fea73f3145dcdd551f09e951007d223d833"} Sep 16 12:48:49.892753 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:49.892685 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:49.994057 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:49.994025 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:50.378733 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:50.378701 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:50.379120 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:50.378841 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 12:48:50.379120 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:50.378914 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls podName:42ae37fd-a147-4072-b59d-ebbda3c45206 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:54.37889574 +0000 UTC m=+41.571544327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls") pod "dns-default-vvb26" (UID: "42ae37fd-a147-4072-b59d-ebbda3c45206") : secret "dns-default-metrics-tls" not found Sep 16 12:48:50.651403 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:50.651175 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bslcs" podStartSLOduration=16.925022125 podStartE2EDuration="37.651141748s" totalImagesPullingTime="20.726119623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:49.66833752 +0000 UTC m=+36.860986127" watchObservedRunningTime="2026-09-16 12:48:50.651141748 +0000 UTC m=+37.843790336" Sep 16 12:48:50.651768 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:50.651678 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-st6jz" podStartSLOduration=2.443882632 podStartE2EDuration="4.65167012s" totalImagesPullingTime="2.207787488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:50.651006997 +0000 UTC m=+37.843655596" watchObservedRunningTime="2026-09-16 12:48:50.65167012 +0000 UTC m=+37.844318702" Sep 16 12:48:51.665634 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:51.665599 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:51.757709 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:51.757682 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:54.403210 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:54.403155 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:48:54.403585 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:54.403263 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 12:48:54.403585 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:48:54.403334 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls podName:42ae37fd-a147-4072-b59d-ebbda3c45206 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:02.403317405 +0000 UTC m=+49.595965977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls") pod "dns-default-vvb26" (UID: "42ae37fd-a147-4072-b59d-ebbda3c45206") : secret "dns-default-metrics-tls" not found Sep 16 12:48:54.525204 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:54.525172 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:54.616461 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:54.616433 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:57.820700 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:57.820669 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:48:57.905890 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:48:57.905856 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:02.451538 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:02.451500 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:49:02.451903 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:02.451639 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 12:49:02.451903 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:02.451703 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls podName:42ae37fd-a147-4072-b59d-ebbda3c45206 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.451686234 +0000 UTC m=+65.644334817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls") pod "dns-default-vvb26" (UID: "42ae37fd-a147-4072-b59d-ebbda3c45206") : secret "dns-default-metrics-tls" not found Sep 16 12:49:05.616576 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.616538 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d"] Sep 16 12:49:05.618452 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.618435 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.621892 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.621870 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-s2xdv\"" Sep 16 12:49:05.621998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.621895 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 16 12:49:05.621998 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.621937 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 16 12:49:05.623265 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.623249 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 16 12:49:05.623360 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.623249 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 16 12:49:05.635239 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.635215 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d"] Sep 16 12:49:05.773872 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.773837 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d27d6dd6-fe80-4842-abfd-a108a94530d3-config\") pod \"service-ca-operator-cf7c776b7-z888d\" (UID: \"d27d6dd6-fe80-4842-abfd-a108a94530d3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.774066 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.773889 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlpnk\" (UniqueName: \"kubernetes.io/projected/d27d6dd6-fe80-4842-abfd-a108a94530d3-kube-api-access-zlpnk\") pod \"service-ca-operator-cf7c776b7-z888d\" (UID: \"d27d6dd6-fe80-4842-abfd-a108a94530d3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.774066 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.773965 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d27d6dd6-fe80-4842-abfd-a108a94530d3-serving-cert\") pod \"service-ca-operator-cf7c776b7-z888d\" (UID: \"d27d6dd6-fe80-4842-abfd-a108a94530d3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.823524 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.823493 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc"] Sep 16 12:49:05.825410 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.825395 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:05.829940 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.829665 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 16 12:49:05.830346 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.830322 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lr46c\"" Sep 16 12:49:05.830706 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.830355 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 16 12:49:05.830706 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.830659 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 16 12:49:05.831027 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.831008 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 16 12:49:05.846829 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.846802 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8stkv"] Sep 16 12:49:05.848766 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.848751 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc"] Sep 16 12:49:05.848841 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.848830 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:05.855856 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.855838 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 16 12:49:05.858331 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.858315 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zjslv\"" Sep 16 12:49:05.859267 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.859252 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 16 12:49:05.859360 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.859276 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 16 12:49:05.864265 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.864244 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 16 12:49:05.867097 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.867048 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 16 12:49:05.875011 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.874990 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d27d6dd6-fe80-4842-abfd-a108a94530d3-config\") pod \"service-ca-operator-cf7c776b7-z888d\" (UID: \"d27d6dd6-fe80-4842-abfd-a108a94530d3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.875106 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.875031 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlpnk\" (UniqueName: \"kubernetes.io/projected/d27d6dd6-fe80-4842-abfd-a108a94530d3-kube-api-access-zlpnk\") pod \"service-ca-operator-cf7c776b7-z888d\" (UID: \"d27d6dd6-fe80-4842-abfd-a108a94530d3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.875106 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.875062 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d27d6dd6-fe80-4842-abfd-a108a94530d3-serving-cert\") pod \"service-ca-operator-cf7c776b7-z888d\" (UID: \"d27d6dd6-fe80-4842-abfd-a108a94530d3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.875560 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.875540 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d27d6dd6-fe80-4842-abfd-a108a94530d3-config\") pod \"service-ca-operator-cf7c776b7-z888d\" (UID: \"d27d6dd6-fe80-4842-abfd-a108a94530d3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.878406 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.878387 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d27d6dd6-fe80-4842-abfd-a108a94530d3-serving-cert\") pod \"service-ca-operator-cf7c776b7-z888d\" (UID: \"d27d6dd6-fe80-4842-abfd-a108a94530d3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.879306 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.879288 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8stkv"] Sep 16 12:49:05.892068 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.892044 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlpnk\" (UniqueName: \"kubernetes.io/projected/d27d6dd6-fe80-4842-abfd-a108a94530d3-kube-api-access-zlpnk\") pod \"service-ca-operator-cf7c776b7-z888d\" (UID: \"d27d6dd6-fe80-4842-abfd-a108a94530d3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.927416 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.927396 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" Sep 16 12:49:05.975634 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.975608 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccb21282-c2dd-405b-9041-83c4ec35c7a3-trusted-ca\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:05.975749 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.975647 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mlp\" (UniqueName: \"kubernetes.io/projected/ccb21282-c2dd-405b-9041-83c4ec35c7a3-kube-api-access-m5mlp\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:05.975749 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.975683 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0dc2888-aafe-47b0-bf94-0ba5fe2238c5-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vd2fc\" (UID: \"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:05.975872 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.975813 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ccb21282-c2dd-405b-9041-83c4ec35c7a3-serving-cert\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:05.975872 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.975858 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sth2r\" (UniqueName: \"kubernetes.io/projected/d0dc2888-aafe-47b0-bf94-0ba5fe2238c5-kube-api-access-sth2r\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vd2fc\" (UID: \"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:05.975982 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.975897 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0dc2888-aafe-47b0-bf94-0ba5fe2238c5-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vd2fc\" (UID: \"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:05.975982 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:05.975924 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ccb21282-c2dd-405b-9041-83c4ec35c7a3-config\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:06.076496 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.076455 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0dc2888-aafe-47b0-bf94-0ba5fe2238c5-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vd2fc\" (UID: \"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:06.076496 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.076495 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ccb21282-c2dd-405b-9041-83c4ec35c7a3-config\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:06.076817 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.076522 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccb21282-c2dd-405b-9041-83c4ec35c7a3-trusted-ca\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:06.076817 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.076541 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mlp\" (UniqueName: \"kubernetes.io/projected/ccb21282-c2dd-405b-9041-83c4ec35c7a3-kube-api-access-m5mlp\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:06.076817 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.076563 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0dc2888-aafe-47b0-bf94-0ba5fe2238c5-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vd2fc\" (UID: \"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:06.076817 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.076599 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ccb21282-c2dd-405b-9041-83c4ec35c7a3-serving-cert\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:06.076817 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.076633 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sth2r\" (UniqueName: \"kubernetes.io/projected/d0dc2888-aafe-47b0-bf94-0ba5fe2238c5-kube-api-access-sth2r\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vd2fc\" (UID: \"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:06.077311 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.077277 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0dc2888-aafe-47b0-bf94-0ba5fe2238c5-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vd2fc\" (UID: \"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:06.077405 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.077386 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ccb21282-c2dd-405b-9041-83c4ec35c7a3-config\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:06.077476 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.077461 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccb21282-c2dd-405b-9041-83c4ec35c7a3-trusted-ca\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:06.078861 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.078841 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0dc2888-aafe-47b0-bf94-0ba5fe2238c5-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vd2fc\" (UID: \"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:06.078949 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.078932 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ccb21282-c2dd-405b-9041-83c4ec35c7a3-serving-cert\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:06.088658 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.088633 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mlp\" (UniqueName: \"kubernetes.io/projected/ccb21282-c2dd-405b-9041-83c4ec35c7a3-kube-api-access-m5mlp\") pod \"console-operator-696884fb9c-8stkv\" (UID: \"ccb21282-c2dd-405b-9041-83c4ec35c7a3\") " pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:06.088931 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.088915 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sth2r\" (UniqueName: \"kubernetes.io/projected/d0dc2888-aafe-47b0-bf94-0ba5fe2238c5-kube-api-access-sth2r\") pod \"kube-storage-version-migrator-operator-6c8f84f897-vd2fc\" (UID: \"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:06.090503 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.090483 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:06.134645 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.134577 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" Sep 16 12:49:06.157822 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.157797 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:06.177419 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.177389 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:06.204213 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.204181 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d"] Sep 16 12:49:06.415807 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.415719 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc"] Sep 16 12:49:06.418495 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:06.418471 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0dc2888_aafe_47b0_bf94_0ba5fe2238c5.slice/crio-7df2b38becc5a7926dbd195e331024f5c1941085b1ea591026f4399d8774c9ac WatchSource:0}: Error finding container 7df2b38becc5a7926dbd195e331024f5c1941085b1ea591026f4399d8774c9ac: Status 404 returned error can't find the container with id 7df2b38becc5a7926dbd195e331024f5c1941085b1ea591026f4399d8774c9ac Sep 16 12:49:06.432283 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.432261 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8stkv"] Sep 16 12:49:06.434771 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:06.434747 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb21282_c2dd_405b_9041_83c4ec35c7a3.slice/crio-18e810211beacce87338f15482bb8d135009b31cee9a6e1610f197d84a9b70ea WatchSource:0}: Error finding container 18e810211beacce87338f15482bb8d135009b31cee9a6e1610f197d84a9b70ea: Status 404 returned error can't find the container with id 18e810211beacce87338f15482bb8d135009b31cee9a6e1610f197d84a9b70ea Sep 16 12:49:06.662412 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.662381 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" event={"ID":"ccb21282-c2dd-405b-9041-83c4ec35c7a3","Type":"ContainerStarted","Data":"18e810211beacce87338f15482bb8d135009b31cee9a6e1610f197d84a9b70ea"} Sep 16 12:49:06.663526 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.663492 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" event={"ID":"d27d6dd6-fe80-4842-abfd-a108a94530d3","Type":"ContainerStarted","Data":"0d2070be31a0569fd834ebf3b9f1108e471217822e96d809762e18740b2ac3d4"} Sep 16 12:49:06.664528 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:06.664508 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" event={"ID":"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5","Type":"ContainerStarted","Data":"7df2b38becc5a7926dbd195e331024f5c1941085b1ea591026f4399d8774c9ac"} Sep 16 12:49:08.377403 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:08.377363 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-npbm5"] Sep 16 12:49:08.379875 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:08.379860 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-npbm5" Sep 16 12:49:08.382710 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:08.382685 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4kp8h\"" Sep 16 12:49:08.393774 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:08.391180 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-npbm5"] Sep 16 12:49:08.495837 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:08.495797 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd648\" (UniqueName: \"kubernetes.io/projected/3cced485-3b23-4c6b-acc3-4d644b0ff0ae-kube-api-access-dd648\") pod \"network-check-source-6f8b768455-npbm5\" (UID: \"3cced485-3b23-4c6b-acc3-4d644b0ff0ae\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-npbm5" Sep 16 12:49:08.596483 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:08.596448 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd648\" (UniqueName: \"kubernetes.io/projected/3cced485-3b23-4c6b-acc3-4d644b0ff0ae-kube-api-access-dd648\") pod \"network-check-source-6f8b768455-npbm5\" (UID: \"3cced485-3b23-4c6b-acc3-4d644b0ff0ae\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-npbm5" Sep 16 12:49:08.606740 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:08.606710 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd648\" (UniqueName: \"kubernetes.io/projected/3cced485-3b23-4c6b-acc3-4d644b0ff0ae-kube-api-access-dd648\") pod \"network-check-source-6f8b768455-npbm5\" (UID: \"3cced485-3b23-4c6b-acc3-4d644b0ff0ae\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-npbm5" Sep 16 12:49:08.691695 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:08.691636 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-npbm5" Sep 16 12:49:09.408135 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.408061 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-npbm5"] Sep 16 12:49:09.411603 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:09.411573 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cced485_3b23_4c6b_acc3_4d644b0ff0ae.slice/crio-f515e67b396e713eea4b91a4e6564e0e51a9e39781c726efd827439f2efb06cd WatchSource:0}: Error finding container f515e67b396e713eea4b91a4e6564e0e51a9e39781c726efd827439f2efb06cd: Status 404 returned error can't find the container with id f515e67b396e713eea4b91a4e6564e0e51a9e39781c726efd827439f2efb06cd Sep 16 12:49:09.675519 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.675435 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" event={"ID":"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5","Type":"ContainerStarted","Data":"00e04d9bba0b255123a55ba55ff18de42512f126d8c05dbd1dc3531033f8e7a3"} Sep 16 12:49:09.677074 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.677055 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/0.log" Sep 16 12:49:09.677210 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.677086 2584 generic.go:356] "Generic (PLEG): container finished" podID="ccb21282-c2dd-405b-9041-83c4ec35c7a3" containerID="9fec49f8d0ab921e381fb78c2a6a4dcea91fa2ad3fc5fd4cfb0319a44bb0f7fa" exitCode=255 Sep 16 12:49:09.677210 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.677110 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" event={"ID":"ccb21282-c2dd-405b-9041-83c4ec35c7a3","Type":"ContainerDied","Data":"9fec49f8d0ab921e381fb78c2a6a4dcea91fa2ad3fc5fd4cfb0319a44bb0f7fa"} Sep 16 12:49:09.677410 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.677388 2584 scope.go:117] "RemoveContainer" containerID="9fec49f8d0ab921e381fb78c2a6a4dcea91fa2ad3fc5fd4cfb0319a44bb0f7fa" Sep 16 12:49:09.679259 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.679000 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" event={"ID":"d27d6dd6-fe80-4842-abfd-a108a94530d3","Type":"ContainerStarted","Data":"eb1d5a2bb2e785becb8e80297ddb01bec0a85ecd5d28533392f206521485cc07"} Sep 16 12:49:09.680310 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.680290 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-npbm5" event={"ID":"3cced485-3b23-4c6b-acc3-4d644b0ff0ae","Type":"ContainerStarted","Data":"f515e67b396e713eea4b91a4e6564e0e51a9e39781c726efd827439f2efb06cd"} Sep 16 12:49:09.694640 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.694599 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" podStartSLOduration=1.972761588 podStartE2EDuration="4.694585679s" totalImagesPullingTime="2.721824091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:09.694237178 +0000 UTC m=+56.886885793" watchObservedRunningTime="2026-09-16 12:49:09.694585679 +0000 UTC m=+56.887234301" Sep 16 12:49:09.711662 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.711607 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" podStartSLOduration=1.784532373 podStartE2EDuration="4.711585646s" totalImagesPullingTime="2.927053273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:09.710610416 +0000 UTC m=+56.903259063" watchObservedRunningTime="2026-09-16 12:49:09.711585646 +0000 UTC m=+56.904234236" Sep 16 12:49:09.920684 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:09.920653 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:10.025122 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:10.025091 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:10.685373 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:10.685335 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 12:49:10.685837 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:10.685821 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/0.log" Sep 16 12:49:10.685898 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:10.685857 2584 generic.go:356] "Generic (PLEG): container finished" podID="ccb21282-c2dd-405b-9041-83c4ec35c7a3" containerID="cb51a3d50cd760050b47ad510ce43f4df216b48982846790c7ba4158a5699926" exitCode=255 Sep 16 12:49:10.685972 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:10.685945 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" event={"ID":"ccb21282-c2dd-405b-9041-83c4ec35c7a3","Type":"ContainerDied","Data":"cb51a3d50cd760050b47ad510ce43f4df216b48982846790c7ba4158a5699926"} Sep 16 12:49:10.686043 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:10.686026 2584 scope.go:117] "RemoveContainer" containerID="9fec49f8d0ab921e381fb78c2a6a4dcea91fa2ad3fc5fd4cfb0319a44bb0f7fa" Sep 16 12:49:10.686445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:10.686257 2584 scope.go:117] "RemoveContainer" containerID="cb51a3d50cd760050b47ad510ce43f4df216b48982846790c7ba4158a5699926" Sep 16 12:49:10.686574 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:10.686448 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8stkv_openshift-console-operator(ccb21282-c2dd-405b-9041-83c4ec35c7a3)\"" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" podUID="ccb21282-c2dd-405b-9041-83c4ec35c7a3" Sep 16 12:49:10.698187 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:10.698145 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:10.815620 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:10.815549 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:10.908679 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:10.908642 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:11.006875 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.006836 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:11.147499 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.147454 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:11.259773 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.259689 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:11.417692 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.417664 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:11.519767 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.519694 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:11.690791 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.690759 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 12:49:11.691271 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.691155 2584 scope.go:117] "RemoveContainer" containerID="cb51a3d50cd760050b47ad510ce43f4df216b48982846790c7ba4158a5699926" Sep 16 12:49:11.691379 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:11.691358 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8stkv_openshift-console-operator(ccb21282-c2dd-405b-9041-83c4ec35c7a3)\"" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" podUID="ccb21282-c2dd-405b-9041-83c4ec35c7a3" Sep 16 12:49:11.802117 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.802031 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2"] Sep 16 12:49:11.805391 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.805366 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2" Sep 16 12:49:11.809095 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.808896 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-m8n48\"" Sep 16 12:49:11.809095 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.809052 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 16 12:49:11.809459 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.809440 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 16 12:49:11.820993 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.820966 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2"] Sep 16 12:49:11.843585 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.843553 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:11.910421 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.910387 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:11.929454 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:11.929426 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqv6\" (UniqueName: \"kubernetes.io/projected/a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16-kube-api-access-bnqv6\") pod \"migrator-8b4d697c5-v5nz2\" (UID: \"a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2" Sep 16 12:49:12.030855 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.030809 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqv6\" (UniqueName: \"kubernetes.io/projected/a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16-kube-api-access-bnqv6\") pod \"migrator-8b4d697c5-v5nz2\" (UID: \"a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2" Sep 16 12:49:12.039952 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.039922 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqv6\" (UniqueName: \"kubernetes.io/projected/a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16-kube-api-access-bnqv6\") pod \"migrator-8b4d697c5-v5nz2\" (UID: \"a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2" Sep 16 12:49:12.116692 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.116611 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2" Sep 16 12:49:12.604572 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.604547 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmzs" Sep 16 12:49:12.623511 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.623446 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jmh42"] Sep 16 12:49:12.625708 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.625690 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.628540 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.628518 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 12:49:12.628649 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.628587 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 12:49:12.629603 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.629570 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x6fmq\"" Sep 16 12:49:12.629603 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.629583 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 12:49:12.629776 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.629633 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 12:49:12.639826 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.639805 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2"] Sep 16 12:49:12.641494 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.641464 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jmh42"] Sep 16 12:49:12.644402 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.644261 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:12.644402 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:12.644352 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97a6fd4_8eea_44e9_b4d6_fc5b7c18ed16.slice/crio-0be19d49dc30bde0338b735351ffa965f76ee4295fe56ca12c35d70ce739368b WatchSource:0}: Error finding container 0be19d49dc30bde0338b735351ffa965f76ee4295fe56ca12c35d70ce739368b: Status 404 returned error can't find the container with id 0be19d49dc30bde0338b735351ffa965f76ee4295fe56ca12c35d70ce739368b Sep 16 12:49:12.695543 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.695509 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2" event={"ID":"a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16","Type":"ContainerStarted","Data":"0be19d49dc30bde0338b735351ffa965f76ee4295fe56ca12c35d70ce739368b"} Sep 16 12:49:12.701290 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.701259 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-npbm5" event={"ID":"3cced485-3b23-4c6b-acc3-4d644b0ff0ae","Type":"ContainerStarted","Data":"36c6fd95dffc8165ebb39601b2ee0c725d5c9f8c9e57e69bb74f5a8fa01d5ff8"} Sep 16 12:49:12.723124 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.723089 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-npbm5" podStartSLOduration=1.726886134 podStartE2EDuration="4.723076521s" totalImagesPullingTime="2.996190387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:12.722668451 +0000 UTC m=+59.915317051" watchObservedRunningTime="2026-09-16 12:49:12.723076521 +0000 UTC m=+59.915725110" Sep 16 12:49:12.736455 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.736433 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-crio-socket\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.736653 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.736620 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.736745 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.736664 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.736745 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.736693 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5csp6\" (UniqueName: \"kubernetes.io/projected/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-api-access-5csp6\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.736837 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.736778 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-data-volume\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.759806 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.759785 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:12.837610 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.837568 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-data-volume\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.837790 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.837626 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-crio-socket\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.837790 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.837683 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.837790 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.837708 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.837790 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.837709 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-crio-socket\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.837790 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.837734 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5csp6\" (UniqueName: \"kubernetes.io/projected/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-api-access-5csp6\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.838073 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:12.837847 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 12:49:12.838073 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:12.837883 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 12:49:12.838073 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.837918 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-data-volume\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.838073 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:12.837928 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm podName:8f1b53d5-d2b5-437e-8944-96aa1dd90fce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:13.337906461 +0000 UTC m=+60.530555033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jmh42" (UID: "8f1b53d5-d2b5-437e-8944-96aa1dd90fce") : configmap "kube-rbac-proxy" not found Sep 16 12:49:12.838073 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:12.837946 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls podName:8f1b53d5-d2b5-437e-8944-96aa1dd90fce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:13.337937243 +0000 UTC m=+60.530585814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jmh42" (UID: "8f1b53d5-d2b5-437e-8944-96aa1dd90fce") : secret "insights-runtime-extractor-tls" not found Sep 16 12:49:12.849892 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.849864 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5csp6\" (UniqueName: \"kubernetes.io/projected/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-api-access-5csp6\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:12.869289 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.869263 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:12.955761 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:12.955688 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:13.024853 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.024817 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-z9vrk"] Sep 16 12:49:13.027133 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.027115 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:13.032663 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:13.032586 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"service-ca-568869fdc5-z9vrk\" is forbidden: User \"system:node:ip-10-0-141-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-service-ca\": no relationship found between node 'ip-10-0-141-139.ec2.internal' and this object" podUID="2d30e8b3-acbc-4c17-a5e4-352db75fa1fe" pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:13.032663 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:13.032597 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"service-ca-dockercfg-qkj76\" is forbidden: User \"system:node:ip-10-0-141-139.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-service-ca\": no relationship found between node 'ip-10-0-141-139.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qkj76\"" type="*v1.Secret" Sep 16 12:49:13.032663 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:13.032609 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-139.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-service-ca\": no relationship found between node 'ip-10-0-141-139.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 16 12:49:13.032663 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:13.032598 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"signing-cabundle\" is forbidden: User \"system:node:ip-10-0-141-139.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-service-ca\": no relationship found between node 'ip-10-0-141-139.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" type="*v1.ConfigMap" Sep 16 12:49:13.032663 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:13.032604 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-139.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-service-ca\": no relationship found between node 'ip-10-0-141-139.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 16 12:49:13.032663 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:13.032601 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"signing-key\" is forbidden: User \"system:node:ip-10-0-141-139.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-service-ca\": no relationship found between node 'ip-10-0-141-139.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca\"/\"signing-key\"" type="*v1.Secret" Sep 16 12:49:13.043963 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.043943 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-z9vrk"] Sep 16 12:49:13.111855 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.111823 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:13.140763 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.140724 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdpdm\" (UniqueName: \"kubernetes.io/projected/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-kube-api-access-pdpdm\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:13.140912 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.140830 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-cabundle\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:13.140912 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.140902 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-key\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:13.226073 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.225993 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:13.242260 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.242231 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpdm\" (UniqueName: \"kubernetes.io/projected/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-kube-api-access-pdpdm\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:13.242411 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.242324 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-cabundle\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:13.242411 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.242395 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-key\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:13.342871 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.342835 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:13.343039 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.342884 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:13.343039 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:13.342933 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 12:49:13.343039 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:13.342995 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm podName:8f1b53d5-d2b5-437e-8944-96aa1dd90fce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:14.342976958 +0000 UTC m=+61.535625559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jmh42" (UID: "8f1b53d5-d2b5-437e-8944-96aa1dd90fce") : configmap "kube-rbac-proxy" not found Sep 16 12:49:13.343039 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:13.343015 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 12:49:13.343286 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:13.343067 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls podName:8f1b53d5-d2b5-437e-8944-96aa1dd90fce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:14.343052168 +0000 UTC m=+61.535700758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jmh42" (UID: "8f1b53d5-d2b5-437e-8944-96aa1dd90fce") : secret "insights-runtime-extractor-tls" not found Sep 16 12:49:13.905318 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:13.905284 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:14.042675 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.042645 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:14.110537 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.110513 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 16 12:49:14.142598 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.142575 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 16 12:49:14.147374 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.147352 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdpdm\" (UniqueName: \"kubernetes.io/projected/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-kube-api-access-pdpdm\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:14.242728 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:14.242699 2584 configmap.go:193] Couldn't get configMap openshift-service-ca/signing-cabundle: failed to sync configmap cache: timed out waiting for the condition Sep 16 12:49:14.242884 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:14.242778 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-cabundle podName:2d30e8b3-acbc-4c17-a5e4-352db75fa1fe nodeName:}" failed. No retries permitted until 2026-09-16 12:49:14.74276267 +0000 UTC m=+61.935411242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "signing-cabundle" (UniqueName: "kubernetes.io/configmap/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-cabundle") pod "service-ca-568869fdc5-z9vrk" (UID: "2d30e8b3-acbc-4c17-a5e4-352db75fa1fe") : failed to sync configmap cache: timed out waiting for the condition Sep 16 12:49:14.242884 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:14.242700 2584 secret.go:189] Couldn't get secret openshift-service-ca/signing-key: failed to sync secret cache: timed out waiting for the condition Sep 16 12:49:14.242884 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:14.242858 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-key podName:2d30e8b3-acbc-4c17-a5e4-352db75fa1fe nodeName:}" failed. No retries permitted until 2026-09-16 12:49:14.74284328 +0000 UTC m=+61.935491852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "signing-key" (UniqueName: "kubernetes.io/secret/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-key") pod "service-ca-568869fdc5-z9vrk" (UID: "2d30e8b3-acbc-4c17-a5e4-352db75fa1fe") : failed to sync secret cache: timed out waiting for the condition Sep 16 12:49:14.350776 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.350698 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:14.350776 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.350731 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:14.350942 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:14.350827 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 12:49:14.350942 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:14.350868 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 12:49:14.350942 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:14.350886 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm podName:8f1b53d5-d2b5-437e-8944-96aa1dd90fce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:16.350872754 +0000 UTC m=+63.543521321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jmh42" (UID: "8f1b53d5-d2b5-437e-8944-96aa1dd90fce") : configmap "kube-rbac-proxy" not found Sep 16 12:49:14.350942 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:14.350905 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls podName:8f1b53d5-d2b5-437e-8944-96aa1dd90fce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:16.350894498 +0000 UTC m=+63.543543081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jmh42" (UID: "8f1b53d5-d2b5-437e-8944-96aa1dd90fce") : secret "insights-runtime-extractor-tls" not found Sep 16 12:49:14.397500 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.397473 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 16 12:49:14.447314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.447287 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qkj76\"" Sep 16 12:49:14.600536 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.600509 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 16 12:49:14.708878 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.708801 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2" event={"ID":"a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16","Type":"ContainerStarted","Data":"758a4fed9352bcda2b2ab929dc73118e716e7b7acf46ef0480afbe9c8d387dc8"} Sep 16 12:49:14.708878 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.708859 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2" event={"ID":"a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16","Type":"ContainerStarted","Data":"be3781d100f8ff8dbfedacdfae1c8cfaae61bd9de456158472f7c06cb361cd91"} Sep 16 12:49:14.726847 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.726807 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-v5nz2" podStartSLOduration=2.660077314 podStartE2EDuration="3.726793886s" totalImagesPullingTime="1.066716572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:14.726216911 +0000 UTC m=+61.918865520" watchObservedRunningTime="2026-09-16 12:49:14.726793886 +0000 UTC m=+61.919442476" Sep 16 12:49:14.753919 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.753891 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-cabundle\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:14.754024 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.753948 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-key\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:14.754575 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.754557 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-cabundle\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:14.756252 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.756223 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2d30e8b3-acbc-4c17-a5e4-352db75fa1fe-signing-key\") pod \"service-ca-568869fdc5-z9vrk\" (UID: \"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe\") " pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:14.836551 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:14.836522 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" Sep 16 12:49:15.107624 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:15.107594 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-z9vrk"] Sep 16 12:49:15.111235 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:15.111210 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d30e8b3_acbc_4c17_a5e4_352db75fa1fe.slice/crio-5f41406353d31f72c9f2d9953caa2881fb0c9bada4017e770f54a733de91ee55 WatchSource:0}: Error finding container 5f41406353d31f72c9f2d9953caa2881fb0c9bada4017e770f54a733de91ee55: Status 404 returned error can't find the container with id 5f41406353d31f72c9f2d9953caa2881fb0c9bada4017e770f54a733de91ee55 Sep 16 12:49:15.189272 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:15.189245 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c64wg_9711ead1-9753-4be1-b0f5-78cd0ac6f89f/dns-node-resolver/0.log" Sep 16 12:49:15.713442 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:15.713397 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" event={"ID":"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe","Type":"ContainerStarted","Data":"74510e3ce3adfcbc656bb2afab80c582394d7f7c2949e03433ac6bdc25033a84"} Sep 16 12:49:15.713654 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:15.713457 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" event={"ID":"2d30e8b3-acbc-4c17-a5e4-352db75fa1fe","Type":"ContainerStarted","Data":"5f41406353d31f72c9f2d9953caa2881fb0c9bada4017e770f54a733de91ee55"} Sep 16 12:49:15.741592 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:15.741543 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-z9vrk" podStartSLOduration=2.74152751 podStartE2EDuration="2.74152751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:15.741040972 +0000 UTC m=+62.933689560" watchObservedRunningTime="2026-09-16 12:49:15.74152751 +0000 UTC m=+62.934176142" Sep 16 12:49:16.158842 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:16.158810 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:16.159318 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:16.158856 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:16.159318 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:16.159239 2584 scope.go:117] "RemoveContainer" containerID="cb51a3d50cd760050b47ad510ce43f4df216b48982846790c7ba4158a5699926" Sep 16 12:49:16.159430 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:16.159383 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8stkv_openshift-console-operator(ccb21282-c2dd-405b-9041-83c4ec35c7a3)\"" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" podUID="ccb21282-c2dd-405b-9041-83c4ec35c7a3" Sep 16 12:49:16.188884 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:16.188863 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gbhf2_b6555a32-a5f6-4d65-82c7-6aaabaff1fbc/node-ca/0.log" Sep 16 12:49:16.367362 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:16.367327 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:16.367362 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:16.367361 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:16.367594 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:16.367441 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 12:49:16.367594 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:16.367471 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 12:49:16.367594 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:16.367507 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm podName:8f1b53d5-d2b5-437e-8944-96aa1dd90fce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:20.367492147 +0000 UTC m=+67.560140713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jmh42" (UID: "8f1b53d5-d2b5-437e-8944-96aa1dd90fce") : configmap "kube-rbac-proxy" not found Sep 16 12:49:16.367594 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:16.367521 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls podName:8f1b53d5-d2b5-437e-8944-96aa1dd90fce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:20.367515553 +0000 UTC m=+67.560164120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jmh42" (UID: "8f1b53d5-d2b5-437e-8944-96aa1dd90fce") : secret "insights-runtime-extractor-tls" not found Sep 16 12:49:16.720602 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:16.720567 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:17.018706 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:17.018617 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:17.175904 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:17.175872 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:17.341920 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:17.341835 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:17.390023 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:17.389952 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-st6jz_49a4be54-687a-4031-9487-72e8d73bdc3d/serve-healthcheck-canary/0.log" Sep 16 12:49:17.453807 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:17.453777 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:17.552874 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:17.552842 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:17.590292 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:17.590256 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-v5nz2_a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16/migrator/0.log" Sep 16 12:49:17.801265 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:17.801239 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-v5nz2_a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16/graceful-termination/0.log" Sep 16 12:49:17.994408 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:17.994381 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-vd2fc_d0dc2888-aafe-47b0-bf94-0ba5fe2238c5/kube-storage-version-migrator-operator/0.log" Sep 16 12:49:18.211743 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:18.211666 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:18.311506 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:18.311478 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:18.485350 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:18.485310 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:49:18.485499 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:18.485473 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 12:49:18.485541 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:18.485536 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls podName:42ae37fd-a147-4072-b59d-ebbda3c45206 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:50.485520758 +0000 UTC m=+97.678169325 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls") pod "dns-default-vvb26" (UID: "42ae37fd-a147-4072-b59d-ebbda3c45206") : secret "dns-default-metrics-tls" not found Sep 16 12:49:18.600871 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:18.600840 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:18.990070 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:18.990042 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:49:18.993488 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:18.993469 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 12:49:19.001113 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:19.001091 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 12:49:19.001188 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:19.001175 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs podName:418a1352-4779-435a-880c-627e3480b111 nodeName:}" failed. No retries permitted until 2026-09-16 12:50:23.001143365 +0000 UTC m=+130.193791961 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs") pod "network-metrics-daemon-jjdcd" (UID: "418a1352-4779-435a-880c-627e3480b111") : secret "metrics-daemon-secret" not found Sep 16 12:49:19.040595 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.040570 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:19.090723 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.090695 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtnp\" (UniqueName: \"kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp\") pod \"network-check-target-btfcd\" (UID: \"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04\") " pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:49:19.090786 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.090741 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:49:19.093140 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.093112 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtnp\" (UniqueName: \"kubernetes.io/projected/aacaaf0a-8a6f-4ced-9c88-1fab1496cb04-kube-api-access-2dtnp\") pod \"network-check-target-btfcd\" (UID: \"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04\") " pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:49:19.093532 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.093517 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 12:49:19.103412 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.103391 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00648523-fd55-498f-b8c6-704f6c9d7e47-original-pull-secret\") pod \"global-pull-secret-syncer-hrjnz\" (UID: \"00648523-fd55-498f-b8c6-704f6c9d7e47\") " pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:49:19.381927 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.381847 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-djtjx\"" Sep 16 12:49:19.389515 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.389499 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:49:19.393314 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.393292 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrjnz" Sep 16 12:49:19.409616 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.409589 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:19.614533 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.614503 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:19.671414 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.671336 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrjnz"] Sep 16 12:49:19.673387 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.673366 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-btfcd"] Sep 16 12:49:19.674634 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:19.674612 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00648523_fd55_498f_b8c6_704f6c9d7e47.slice/crio-453f0b9c752167514900467e2848b1991bc7638616ec87144901c15f4c8fa064 WatchSource:0}: Error finding container 453f0b9c752167514900467e2848b1991bc7638616ec87144901c15f4c8fa064: Status 404 returned error can't find the container with id 453f0b9c752167514900467e2848b1991bc7638616ec87144901c15f4c8fa064 Sep 16 12:49:19.675792 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:19.675764 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacaaf0a_8a6f_4ced_9c88_1fab1496cb04.slice/crio-7c75d3d9e78dde7ce5591ce32db370e34c3b28c4cb0605d40e182dcacd1c552c WatchSource:0}: Error finding container 7c75d3d9e78dde7ce5591ce32db370e34c3b28c4cb0605d40e182dcacd1c552c: Status 404 returned error can't find the container with id 7c75d3d9e78dde7ce5591ce32db370e34c3b28c4cb0605d40e182dcacd1c552c Sep 16 12:49:19.730347 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.730282 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-btfcd" event={"ID":"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04","Type":"ContainerStarted","Data":"7c75d3d9e78dde7ce5591ce32db370e34c3b28c4cb0605d40e182dcacd1c552c"} Sep 16 12:49:19.731486 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:19.731460 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrjnz" event={"ID":"00648523-fd55-498f-b8c6-704f6c9d7e47","Type":"ContainerStarted","Data":"453f0b9c752167514900467e2848b1991bc7638616ec87144901c15f4c8fa064"} Sep 16 12:49:20.002170 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:20.002129 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:20.182716 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:20.182681 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:20.257136 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:20.257045 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:20.402150 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:20.402120 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:20.402150 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:20.402154 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:20.402618 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:20.402258 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 12:49:20.402618 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:20.402307 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 12:49:20.402618 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:20.402341 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm podName:8f1b53d5-d2b5-437e-8944-96aa1dd90fce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:28.402317294 +0000 UTC m=+75.594965864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jmh42" (UID: "8f1b53d5-d2b5-437e-8944-96aa1dd90fce") : configmap "kube-rbac-proxy" not found Sep 16 12:49:20.402618 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:20.402361 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls podName:8f1b53d5-d2b5-437e-8944-96aa1dd90fce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:28.402351439 +0000 UTC m=+75.595000011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jmh42" (UID: "8f1b53d5-d2b5-437e-8944-96aa1dd90fce") : secret "insights-runtime-extractor-tls" not found Sep 16 12:49:20.612572 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:20.612486 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:20.736532 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:20.736498 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-btfcd" event={"ID":"aacaaf0a-8a6f-4ced-9c88-1fab1496cb04","Type":"ContainerStarted","Data":"b0e7660ca9567b6d89e0222cb6a6b6a1585d8c65f17b7217cbc70117e32f532e"} Sep 16 12:49:20.736715 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:20.736635 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:49:20.757333 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:20.757280 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-btfcd" podStartSLOduration=67.757257653 podStartE2EDuration="1m7.757257653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:20.756545037 +0000 UTC m=+67.949193651" watchObservedRunningTime="2026-09-16 12:49:20.757257653 +0000 UTC m=+67.949906243" Sep 16 12:49:20.916041 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:20.915961 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:21.008882 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:21.008850 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:21.198058 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:21.197970 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:21.258800 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:21.258765 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:21.856773 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:21.856721 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:22.201633 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:22.201558 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:22.402665 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:22.402634 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:23.226537 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:23.226496 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:23.389650 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:23.389598 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:23.490891 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:23.490865 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:23.678937 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:23.678904 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:23.748726 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:23.748642 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrjnz" event={"ID":"00648523-fd55-498f-b8c6-704f6c9d7e47","Type":"ContainerStarted","Data":"7936c29f40c06c8949587b29aba303676ffb641f7f2b951dd2df7c080dfc1951"} Sep 16 12:49:23.772321 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:23.772271 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hrjnz" podStartSLOduration=67.038460213 podStartE2EDuration="1m10.772256877s" totalImagesPullingTime="3.733796664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:23.77146979 +0000 UTC m=+70.964118380" watchObservedRunningTime="2026-09-16 12:49:23.772256877 +0000 UTC m=+70.964905465" Sep 16 12:49:24.000073 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:23.999973 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:24.121270 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:24.121241 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:24.608942 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:24.608910 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:25.229241 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:25.229210 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:25.408933 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:25.408900 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:25.800255 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:25.800222 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:26.024506 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:26.024480 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:26.688722 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:26.688688 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:26.835930 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:26.835902 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:27.255227 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:27.255194 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:27.480648 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:27.480618 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:28.458262 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:28.458210 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:28.458262 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:28.458259 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:28.458729 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:28.458706 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:28.460496 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:28.460475 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f1b53d5-d2b5-437e-8944-96aa1dd90fce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmh42\" (UID: \"8f1b53d5-d2b5-437e-8944-96aa1dd90fce\") " pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:28.539516 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:28.539486 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x6fmq\"" Sep 16 12:49:28.546866 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:28.546850 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jmh42" Sep 16 12:49:28.833363 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:28.833336 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jmh42"] Sep 16 12:49:28.838412 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:28.838387 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1b53d5_d2b5_437e_8944_96aa1dd90fce.slice/crio-753315784aab07a9aaa40c984cbd56a2b1073e77e48a3e3f837a070159431660 WatchSource:0}: Error finding container 753315784aab07a9aaa40c984cbd56a2b1073e77e48a3e3f837a070159431660: Status 404 returned error can't find the container with id 753315784aab07a9aaa40c984cbd56a2b1073e77e48a3e3f837a070159431660 Sep 16 12:49:29.770185 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:29.770132 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmh42" event={"ID":"8f1b53d5-d2b5-437e-8944-96aa1dd90fce","Type":"ContainerStarted","Data":"db9151c460c48330ca3ef96b10fa90e5b4c7982dd337ddee402fe989d509f463"} Sep 16 12:49:29.770549 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:29.770205 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmh42" event={"ID":"8f1b53d5-d2b5-437e-8944-96aa1dd90fce","Type":"ContainerStarted","Data":"715a9be9071ac366af490d6fa30f9beafbbbb5d620be04932ac08d805b22b115"} Sep 16 12:49:29.770549 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:29.770224 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmh42" event={"ID":"8f1b53d5-d2b5-437e-8944-96aa1dd90fce","Type":"ContainerStarted","Data":"753315784aab07a9aaa40c984cbd56a2b1073e77e48a3e3f837a070159431660"} Sep 16 12:49:30.467957 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:30.467929 2584 scope.go:117] "RemoveContainer" containerID="cb51a3d50cd760050b47ad510ce43f4df216b48982846790c7ba4158a5699926" Sep 16 12:49:30.775744 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:30.775673 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 12:49:30.776211 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:30.775773 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" event={"ID":"ccb21282-c2dd-405b-9041-83c4ec35c7a3","Type":"ContainerStarted","Data":"f6a8f41c1edcca35525b7be8936ceacd5a351302235595152b03802079145866"} Sep 16 12:49:30.776211 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:30.776203 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:30.794965 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:30.794912 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" podStartSLOduration=23.090576917 podStartE2EDuration="25.794891959s" totalImagesPullingTime="2.704315042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:30.793940355 +0000 UTC m=+77.986588967" watchObservedRunningTime="2026-09-16 12:49:30.794891959 +0000 UTC m=+77.987540551" Sep 16 12:49:31.436505 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:31.436433 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-8stkv" Sep 16 12:49:31.633960 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:31.633927 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:31.730982 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:31.730956 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:31.780776 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:31.780744 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmh42" event={"ID":"8f1b53d5-d2b5-437e-8944-96aa1dd90fce","Type":"ContainerStarted","Data":"5c5c033c5fe83a35da8d23a25df6a2b0ebdcebf7d37ef87879ed3598bc7b2cab"} Sep 16 12:49:31.802534 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:31.802494 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jmh42" podStartSLOduration=17.740874463 podStartE2EDuration="19.802480941s" totalImagesPullingTime="2.061606478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:49:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:31.800520607 +0000 UTC m=+78.993169209" watchObservedRunningTime="2026-09-16 12:49:31.802480941 +0000 UTC m=+78.995129529" Sep 16 12:49:31.806136 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:31.806115 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:31.955366 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:31.955333 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:32.110017 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:32.109935 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:32.217398 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:32.217360 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:32.505866 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:32.505837 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:32.586270 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:32.586239 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:32.698929 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:32.698900 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:32.779484 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:32.779405 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:33.109022 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:33.108943 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:33.316284 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:33.316251 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:33.559379 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:33.559353 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:34.529668 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:34.529640 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:39.081456 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:39.081345 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:39.238863 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:39.238825 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:39.338200 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:39.338109 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:39.433239 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:39.433207 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:40.917631 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:40.917599 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:40.995408 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:40.995378 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:41.083702 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.083672 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:41.083702 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.083687 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk"] Sep 16 12:49:41.114857 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.114828 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk"] Sep 16 12:49:41.114995 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.114932 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" Sep 16 12:49:41.118553 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.118533 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9csp8\"" Sep 16 12:49:41.118688 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.118568 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 16 12:49:41.149691 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.149664 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/661e33e1-a996-4039-9f50-3a7e4b16ebd7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-g5qkk\" (UID: \"661e33e1-a996-4039-9f50-3a7e4b16ebd7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" Sep 16 12:49:41.250648 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.250614 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/661e33e1-a996-4039-9f50-3a7e4b16ebd7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-g5qkk\" (UID: \"661e33e1-a996-4039-9f50-3a7e4b16ebd7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" Sep 16 12:49:41.250823 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:41.250787 2584 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 16 12:49:41.250895 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:41.250870 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661e33e1-a996-4039-9f50-3a7e4b16ebd7-tls-certificates podName:661e33e1-a996-4039-9f50-3a7e4b16ebd7 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:41.750849001 +0000 UTC m=+88.943497569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/661e33e1-a996-4039-9f50-3a7e4b16ebd7-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-g5qkk" (UID: "661e33e1-a996-4039-9f50-3a7e4b16ebd7") : secret "prometheus-operator-admission-webhook-tls" not found Sep 16 12:49:41.351430 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.351397 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:41.754561 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.754527 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/661e33e1-a996-4039-9f50-3a7e4b16ebd7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-g5qkk\" (UID: \"661e33e1-a996-4039-9f50-3a7e4b16ebd7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" Sep 16 12:49:41.756992 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:41.756969 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/661e33e1-a996-4039-9f50-3a7e4b16ebd7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-g5qkk\" (UID: \"661e33e1-a996-4039-9f50-3a7e4b16ebd7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" Sep 16 12:49:42.025292 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:42.025195 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" Sep 16 12:49:42.294888 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:42.294819 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk"] Sep 16 12:49:42.298101 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:42.298074 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661e33e1_a996_4039_9f50_3a7e4b16ebd7.slice/crio-4da5da9a5b65acee8103581011d6fcacb1ad16150d90ccb12f0ba5ba438f07f4 WatchSource:0}: Error finding container 4da5da9a5b65acee8103581011d6fcacb1ad16150d90ccb12f0ba5ba438f07f4: Status 404 returned error can't find the container with id 4da5da9a5b65acee8103581011d6fcacb1ad16150d90ccb12f0ba5ba438f07f4 Sep 16 12:49:42.817514 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:42.817474 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" event={"ID":"661e33e1-a996-4039-9f50-3a7e4b16ebd7","Type":"ContainerStarted","Data":"4da5da9a5b65acee8103581011d6fcacb1ad16150d90ccb12f0ba5ba438f07f4"} Sep 16 12:49:43.821719 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:43.821678 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" event={"ID":"661e33e1-a996-4039-9f50-3a7e4b16ebd7","Type":"ContainerStarted","Data":"b0d01ecc459273142c22a43da9b8bd30e7b4892a726fdb43986c987cd24f54d1"} Sep 16 12:49:43.822197 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:43.821885 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" Sep 16 12:49:43.826297 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:43.826277 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" Sep 16 12:49:43.839897 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:43.839858 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-g5qkk" podStartSLOduration=1.7963258610000001 podStartE2EDuration="2.83984583s" totalImagesPullingTime="1.043519969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:43.838601011 +0000 UTC m=+91.031249601" watchObservedRunningTime="2026-09-16 12:49:43.83984583 +0000 UTC m=+91.032494457" Sep 16 12:49:44.137883 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:44.137802 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:44.212175 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:44.212133 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:45.080334 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:45.080305 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:47.785311 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:47.785279 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:48.359447 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:48.359414 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:48.461721 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:48.461690 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:50.264201 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.264150 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:50.473302 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.473272 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:50.521732 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.521697 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d"] Sep 16 12:49:50.522964 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.522939 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:49:50.525603 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.525587 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.526352 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.526045 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m9qss"] Sep 16 12:49:50.529034 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.528368 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4vjxr\"" Sep 16 12:49:50.529034 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.528460 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.529034 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.528644 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 12:49:50.529034 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.528704 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 12:49:50.529034 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.528856 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 16 12:49:50.529782 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.529623 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 16 12:49:50.529782 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.529658 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 16 12:49:50.529782 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.529768 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 12:49:50.532701 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.532477 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 12:49:50.532701 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.532480 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ae37fd-a147-4072-b59d-ebbda3c45206-metrics-tls\") pod \"dns-default-vvb26\" (UID: \"42ae37fd-a147-4072-b59d-ebbda3c45206\") " pod="openshift-dns/dns-default-vvb26" Sep 16 12:49:50.533631 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.533245 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 12:49:50.533631 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.533477 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 12:49:50.534141 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.533978 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dd2m8\"" Sep 16 12:49:50.535508 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.535484 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d"] Sep 16 12:49:50.580432 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.580409 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sl6fp\"" Sep 16 12:49:50.588465 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.588432 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vvb26" Sep 16 12:49:50.592661 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.592636 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:50.623369 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623343 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.623507 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623375 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-sys\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.623507 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623397 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11aa326-cac9-419e-9382-7e34a43fd46b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.623507 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623416 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-tls\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.623507 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623433 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d11aa326-cac9-419e-9382-7e34a43fd46b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.623507 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623467 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-accelerators-collector-config\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.623507 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623487 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.623786 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623524 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.623786 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623550 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbxbw\" (UniqueName: \"kubernetes.io/projected/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-kube-api-access-rbxbw\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.623786 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623569 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlxxp\" (UniqueName: \"kubernetes.io/projected/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-api-access-hlxxp\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.623786 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623584 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-metrics-client-ca\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.623786 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623600 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-textfile\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.623786 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623618 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.623786 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623642 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-root\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.623786 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.623656 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-wtmp\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.687822 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.687797 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:50.724067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.723989 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-accelerators-collector-config\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.724067 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724033 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.724280 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724068 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.724280 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724099 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbxbw\" (UniqueName: \"kubernetes.io/projected/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-kube-api-access-rbxbw\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.724280 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724125 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlxxp\" (UniqueName: \"kubernetes.io/projected/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-api-access-hlxxp\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.724280 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724149 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-metrics-client-ca\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.724280 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724197 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-textfile\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.724280 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724224 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.724280 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724259 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-root\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.724618 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724283 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-wtmp\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.724618 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724315 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.724618 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724344 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-sys\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.724618 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724375 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11aa326-cac9-419e-9382-7e34a43fd46b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.724618 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724401 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-tls\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.724618 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724428 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d11aa326-cac9-419e-9382-7e34a43fd46b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.725554 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.724855 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d11aa326-cac9-419e-9382-7e34a43fd46b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.725805 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-root\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:50.725919 2584 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:50.725984 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-tls podName:d11aa326-cac9-419e-9382-7e34a43fd46b nodeName:}" failed. No retries permitted until 2026-09-16 12:49:51.225965899 +0000 UTC m=+98.418614485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-cns4d" (UID: "d11aa326-cac9-419e-9382-7e34a43fd46b") : secret "kube-state-metrics-tls" not found Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.725982 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-textfile\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:50.726290 2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:49:50.726339 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-tls podName:59664ff4-a68b-41fa-b0ee-4b7f85aa41da nodeName:}" failed. No retries permitted until 2026-09-16 12:49:51.226322721 +0000 UTC m=+98.418971321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-tls") pod "node-exporter-m9qss" (UID: "59664ff4-a68b-41fa-b0ee-4b7f85aa41da") : secret "node-exporter-tls" not found Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.726751 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11aa326-cac9-419e-9382-7e34a43fd46b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.726776 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.726790 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-metrics-client-ca\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.726860 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-sys\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.726888 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.726877 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-wtmp\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.727479 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.726983 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-accelerators-collector-config\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.730486 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.730111 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.730486 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.730440 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.739499 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.739478 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbxbw\" (UniqueName: \"kubernetes.io/projected/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-kube-api-access-rbxbw\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:50.746199 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.746140 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlxxp\" (UniqueName: \"kubernetes.io/projected/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-api-access-hlxxp\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:50.787942 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.787911 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:50.865383 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.865356 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vvb26"] Sep 16 12:49:50.868401 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:50.868380 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ae37fd_a147_4072_b59d_ebbda3c45206.slice/crio-4186d995f7cd847afa74763a405a6b62343447da5d3debb48011861a6a957697 WatchSource:0}: Error finding container 4186d995f7cd847afa74763a405a6b62343447da5d3debb48011861a6a957697: Status 404 returned error can't find the container with id 4186d995f7cd847afa74763a405a6b62343447da5d3debb48011861a6a957697 Sep 16 12:49:50.871778 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.871745 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:50.937064 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.937038 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:50.994937 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:50.994823 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:51.230919 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.230885 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-tls\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:51.231099 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.230971 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:51.233362 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.233333 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59664ff4-a68b-41fa-b0ee-4b7f85aa41da-node-exporter-tls\") pod \"node-exporter-m9qss\" (UID: \"59664ff4-a68b-41fa-b0ee-4b7f85aa41da\") " pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:51.233753 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.233730 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11aa326-cac9-419e-9382-7e34a43fd46b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cns4d\" (UID: \"d11aa326-cac9-419e-9382-7e34a43fd46b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:51.448545 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.448461 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" Sep 16 12:49:51.455744 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.455717 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m9qss" Sep 16 12:49:51.466374 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:51.466346 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59664ff4_a68b_41fa_b0ee_4b7f85aa41da.slice/crio-d65aafb5c77a50db8a8662f98ae602095e1959ff4ff64bdd6876423abdfada83 WatchSource:0}: Error finding container d65aafb5c77a50db8a8662f98ae602095e1959ff4ff64bdd6876423abdfada83: Status 404 returned error can't find the container with id d65aafb5c77a50db8a8662f98ae602095e1959ff4ff64bdd6876423abdfada83 Sep 16 12:49:51.486220 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.486125 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:51.582344 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.582315 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:51.600864 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.600838 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 12:49:51.604924 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.604904 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.608421 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.608355 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 16 12:49:51.608761 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.608740 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 16 12:49:51.608942 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.608926 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 16 12:49:51.609099 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.609081 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 16 12:49:51.609324 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.609294 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tng8w\"" Sep 16 12:49:51.609415 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.609345 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 16 12:49:51.609415 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.609387 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 16 12:49:51.609523 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.609489 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 16 12:49:51.609613 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.609594 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 16 12:49:51.610705 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.610446 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 16 12:49:51.623351 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.623326 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 12:49:51.689175 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.689125 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:51.733827 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.733758 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d"] Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736171 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736221 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-web-config\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736265 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-config-out\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736294 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736320 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736345 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736370 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736431 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll5sq\" (UniqueName: \"kubernetes.io/projected/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-kube-api-access-ll5sq\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736464 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736500 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736524 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.736600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736555 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.737267 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.736645 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.737561 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:51.737525 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11aa326_cac9_419e_9382_7e34a43fd46b.slice/crio-7e69d9c4c43d0af95ec06d57f7a5b64f0e37609d4f571520716f5ca57912c686 WatchSource:0}: Error finding container 7e69d9c4c43d0af95ec06d57f7a5b64f0e37609d4f571520716f5ca57912c686: Status 404 returned error can't find the container with id 7e69d9c4c43d0af95ec06d57f7a5b64f0e37609d4f571520716f5ca57912c686 Sep 16 12:49:51.743065 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.742539 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-btfcd" Sep 16 12:49:51.837680 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.837646 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll5sq\" (UniqueName: \"kubernetes.io/projected/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-kube-api-access-ll5sq\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.837832 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.837711 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.837832 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.837759 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.837932 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.837911 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.837978 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.837960 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.838029 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.838006 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.838080 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.838046 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.838080 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.838064 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-web-config\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.838208 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.838092 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-config-out\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.838208 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.838119 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.838208 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.838137 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.838208 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.838144 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.838397 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.838219 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.838397 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.838260 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.839498 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.839449 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.839836 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.839810 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.840899 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.840851 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.842212 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.842188 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-config-out\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.842338 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.842295 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.842536 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.842491 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-web-config\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.843188 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.842795 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.843188 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.842863 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.843188 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.843120 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.843506 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.843227 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.844346 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.844303 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.846316 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.846263 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll5sq\" (UniqueName: \"kubernetes.io/projected/c1d7c3b9-66ea-4d17-a0ed-e95a147502b3-kube-api-access-ll5sq\") pod \"alertmanager-main-0\" (UID: \"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:51.848103 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.848068 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m9qss" event={"ID":"59664ff4-a68b-41fa-b0ee-4b7f85aa41da","Type":"ContainerStarted","Data":"d65aafb5c77a50db8a8662f98ae602095e1959ff4ff64bdd6876423abdfada83"} Sep 16 12:49:51.849520 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.849494 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvb26" event={"ID":"42ae37fd-a147-4072-b59d-ebbda3c45206","Type":"ContainerStarted","Data":"4186d995f7cd847afa74763a405a6b62343447da5d3debb48011861a6a957697"} Sep 16 12:49:51.850856 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.850833 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" event={"ID":"d11aa326-cac9-419e-9382-7e34a43fd46b","Type":"ContainerStarted","Data":"7e69d9c4c43d0af95ec06d57f7a5b64f0e37609d4f571520716f5ca57912c686"} Sep 16 12:49:51.898172 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.898127 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:51.920798 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:51.920775 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 12:49:52.857183 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:52.857081 2584 generic.go:356] "Generic (PLEG): container finished" podID="59664ff4-a68b-41fa-b0ee-4b7f85aa41da" containerID="09f0b02b3f484a685ae5b5b581f06bcfa6838a4cdcb2978ab6025a3548f7c62b" exitCode=0 Sep 16 12:49:52.857552 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:52.857196 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m9qss" event={"ID":"59664ff4-a68b-41fa-b0ee-4b7f85aa41da","Type":"ContainerDied","Data":"09f0b02b3f484a685ae5b5b581f06bcfa6838a4cdcb2978ab6025a3548f7c62b"} Sep 16 12:49:52.859622 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:52.859594 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvb26" event={"ID":"42ae37fd-a147-4072-b59d-ebbda3c45206","Type":"ContainerStarted","Data":"678d8493a04036595157cbcfd515b8a49e95e6b4b6962baced7f4910969cf3a4"} Sep 16 12:49:52.906433 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:52.906404 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 12:49:52.909145 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:49:52.909116 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d7c3b9_66ea_4d17_a0ed_e95a147502b3.slice/crio-16201175ce7421bb1e39d468d5f8b0c2fb000777756ae19f64196f22ecec9f9d WatchSource:0}: Error finding container 16201175ce7421bb1e39d468d5f8b0c2fb000777756ae19f64196f22ecec9f9d: Status 404 returned error can't find the container with id 16201175ce7421bb1e39d468d5f8b0c2fb000777756ae19f64196f22ecec9f9d Sep 16 12:49:53.440105 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.440071 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:53.552757 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.552724 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:53.697970 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.697927 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:53.802612 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.802578 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:53.867145 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.867053 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:53.868795 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.868220 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m9qss" event={"ID":"59664ff4-a68b-41fa-b0ee-4b7f85aa41da","Type":"ContainerStarted","Data":"aea29e3b742cde7f2abdbe5a4a6fd33a3984cb21cfb6adf79d95a1588aa730da"} Sep 16 12:49:53.868795 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.868278 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m9qss" event={"ID":"59664ff4-a68b-41fa-b0ee-4b7f85aa41da","Type":"ContainerStarted","Data":"9080554c49aa0cdfed12b575bc4fc0b60d7d82283547f15dcdb060d67751470f"} Sep 16 12:49:53.871061 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.870846 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvb26" event={"ID":"42ae37fd-a147-4072-b59d-ebbda3c45206","Type":"ContainerStarted","Data":"52d0591d7625694ef68db72f0727f8c5db5ef5511ac6525bf0d71a2d9732ce43"} Sep 16 12:49:53.871061 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.870899 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vvb26" Sep 16 12:49:53.875027 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.875000 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" event={"ID":"d11aa326-cac9-419e-9382-7e34a43fd46b","Type":"ContainerStarted","Data":"f44f9d5f84fba10c65c131bb4d81e5c30cc0b1ba67a1a9dd032e4bb34ac3a73d"} Sep 16 12:49:53.875134 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.875053 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" event={"ID":"d11aa326-cac9-419e-9382-7e34a43fd46b","Type":"ContainerStarted","Data":"8a9a6f129e5238eae628fd82fffa20f70d8b569eb8a538cb3644c13c56dd98af"} Sep 16 12:49:53.875134 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.875083 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" event={"ID":"d11aa326-cac9-419e-9382-7e34a43fd46b","Type":"ContainerStarted","Data":"23b4c7800c74fa73220a5c9970cad56bddc61369b04c7774868742c1154f7cff"} Sep 16 12:49:53.876652 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.876628 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3","Type":"ContainerStarted","Data":"16201175ce7421bb1e39d468d5f8b0c2fb000777756ae19f64196f22ecec9f9d"} Sep 16 12:49:53.896071 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.895977 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m9qss" podStartSLOduration=2.65313125 podStartE2EDuration="3.895959911s" totalImagesPullingTime="1.156814343s" totalInitContainerRuntime="86.014318ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:53.895923421 +0000 UTC m=+101.088572013" watchObservedRunningTime="2026-09-16 12:49:53.895959911 +0000 UTC m=+101.088608501" Sep 16 12:49:53.942559 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.942498 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vvb26" podStartSLOduration=66.186302254 podStartE2EDuration="1m7.942478783s" totalImagesPullingTime="1.756176529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:53.920405594 +0000 UTC m=+101.113054203" watchObservedRunningTime="2026-09-16 12:49:53.942478783 +0000 UTC m=+101.135127373" Sep 16 12:49:53.943505 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.943470 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cns4d" podStartSLOduration=2.366025993 podStartE2EDuration="3.94345656s" totalImagesPullingTime="1.577430567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:53.941831609 +0000 UTC m=+101.134480232" watchObservedRunningTime="2026-09-16 12:49:53.94345656 +0000 UTC m=+101.136105150" Sep 16 12:49:53.946336 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:53.946308 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:54.248257 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:54.248224 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:54.881536 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:54.881436 2584 generic.go:356] "Generic (PLEG): container finished" podID="c1d7c3b9-66ea-4d17-a0ed-e95a147502b3" containerID="0fd8ff2fb9dc7c712dc8a9ac182200d52a9547d01075d1bb156bc3604be2e014" exitCode=0 Sep 16 12:49:54.881973 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:54.881539 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3","Type":"ContainerDied","Data":"0fd8ff2fb9dc7c712dc8a9ac182200d52a9547d01075d1bb156bc3604be2e014"} Sep 16 12:49:55.312931 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:55.312899 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:55.441903 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:55.441833 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:55.956387 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:55.956356 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:56.699999 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:56.699957 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:56.892319 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:56.892235 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3","Type":"ContainerStarted","Data":"41cd4ac741aa8d24c50c1c8841a4230a3fe69ec575a452b0a6ece8ef87d0cc75"} Sep 16 12:49:56.892445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:56.892328 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3","Type":"ContainerStarted","Data":"6f75d0f539ace8d32a0c8ea752e01ab3957c72971e8cc3e902455ec820a1fbc4"} Sep 16 12:49:56.892445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:56.892354 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3","Type":"ContainerStarted","Data":"96c890c91fee3be64333f5fc7955f9dce2e4e32ab73a529ad69e329b2c89a0af"} Sep 16 12:49:56.892445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:56.892378 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3","Type":"ContainerStarted","Data":"2bfc3ee285c0cdd02c58598e34041b6a9df9dff53c63c82924998e6ddbf09a05"} Sep 16 12:49:56.892445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:56.892404 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3","Type":"ContainerStarted","Data":"3f3b1e25ff63031c7a1ed2418fc0f600eee4a6eec1c8d306914c39f445a06a0d"} Sep 16 12:49:57.899594 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:57.899558 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d7c3b9-66ea-4d17-a0ed-e95a147502b3","Type":"ContainerStarted","Data":"57390030118de8d913f2e1fe65d8f806fc6e9792d3ea71ebb5658752b89f7d9f"} Sep 16 12:49:57.942515 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:57.942457 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.32672431 podStartE2EDuration="6.94243934s" totalImagesPullingTime="3.59521209s" totalInitContainerRuntime="20.50294ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:57.941930099 +0000 UTC m=+105.134578687" watchObservedRunningTime="2026-09-16 12:49:57.94243934 +0000 UTC m=+105.135087928" Sep 16 12:49:57.963019 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:57.962989 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:49:58.054294 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:49:58.054261 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:02.080664 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:02.080633 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:02.223720 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:02.223690 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:02.681512 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:02.681480 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:02.745672 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:02.745644 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:03.884096 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:03.884069 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vvb26" Sep 16 12:50:06.808738 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:06.808693 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:06.869141 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:06.869112 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:07.031356 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:07.031326 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:07.140883 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:07.140812 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:08.948898 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:08.948861 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:09.054257 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:09.054226 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:09.152557 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:09.152525 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:09.252515 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:09.252486 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:09.322095 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:09.322065 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:13.087870 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:13.087837 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:13.185742 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:13.185707 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:16.114419 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:16.114389 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:16.215518 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:16.215486 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:21.153099 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:21.153062 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:21.253175 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:21.253123 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:21.313708 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:21.313674 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:23.090432 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:23.090390 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:50:23.092647 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:23.092622 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/418a1352-4779-435a-880c-627e3480b111-metrics-certs\") pod \"network-metrics-daemon-jjdcd\" (UID: \"418a1352-4779-435a-880c-627e3480b111\") " pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:50:23.290195 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:23.290150 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-27x5j\"" Sep 16 12:50:23.296545 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:23.296525 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jjdcd" Sep 16 12:50:23.568921 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:23.568897 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jjdcd"] Sep 16 12:50:23.571370 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:50:23.571340 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418a1352_4779_435a_880c_627e3480b111.slice/crio-5f7d9c1d5da3ccfdf963eac7e1b44467c4946755eccbf2861f8aac1f932b7e41 WatchSource:0}: Error finding container 5f7d9c1d5da3ccfdf963eac7e1b44467c4946755eccbf2861f8aac1f932b7e41: Status 404 returned error can't find the container with id 5f7d9c1d5da3ccfdf963eac7e1b44467c4946755eccbf2861f8aac1f932b7e41 Sep 16 12:50:23.992445 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:23.992417 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jjdcd" event={"ID":"418a1352-4779-435a-880c-627e3480b111","Type":"ContainerStarted","Data":"5f7d9c1d5da3ccfdf963eac7e1b44467c4946755eccbf2861f8aac1f932b7e41"} Sep 16 12:50:24.003113 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:24.003090 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:24.057268 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:24.057245 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:24.997751 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:24.997727 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jjdcd" event={"ID":"418a1352-4779-435a-880c-627e3480b111","Type":"ContainerStarted","Data":"c7ab71c2714148fb191930706ca7312ed891cdb7419ba01bb932b8a470bbbb02"} Sep 16 12:50:26.002549 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:26.002507 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jjdcd" event={"ID":"418a1352-4779-435a-880c-627e3480b111","Type":"ContainerStarted","Data":"68e3d1e5641c86e41229ce51ed0bc5b562f4a65d346643e0fd0b7544be699b73"} Sep 16 12:50:26.021193 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:26.021134 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jjdcd" podStartSLOduration=131.756916703 podStartE2EDuration="2m13.021120457s" totalImagesPullingTime="1.264203754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:26.020197574 +0000 UTC m=+133.212846174" watchObservedRunningTime="2026-09-16 12:50:26.021120457 +0000 UTC m=+133.213769055" Sep 16 12:50:26.129742 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:26.129714 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:26.188222 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:26.188195 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:30.018524 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:30.018497 2584 generic.go:356] "Generic (PLEG): container finished" podID="d27d6dd6-fe80-4842-abfd-a108a94530d3" containerID="eb1d5a2bb2e785becb8e80297ddb01bec0a85ecd5d28533392f206521485cc07" exitCode=0 Sep 16 12:50:30.018880 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:30.018545 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" event={"ID":"d27d6dd6-fe80-4842-abfd-a108a94530d3","Type":"ContainerDied","Data":"eb1d5a2bb2e785becb8e80297ddb01bec0a85ecd5d28533392f206521485cc07"} Sep 16 12:50:30.018880 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:30.018854 2584 scope.go:117] "RemoveContainer" containerID="eb1d5a2bb2e785becb8e80297ddb01bec0a85ecd5d28533392f206521485cc07" Sep 16 12:50:31.025128 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:31.025093 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-z888d" event={"ID":"d27d6dd6-fe80-4842-abfd-a108a94530d3","Type":"ContainerStarted","Data":"3d4263166d4eb4386d9ca4e353371ab831ed6e39813b1485961ee5d64d78ec4c"} Sep 16 12:50:35.041678 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:35.041649 2584 generic.go:356] "Generic (PLEG): container finished" podID="d0dc2888-aafe-47b0-bf94-0ba5fe2238c5" containerID="00e04d9bba0b255123a55ba55ff18de42512f126d8c05dbd1dc3531033f8e7a3" exitCode=0 Sep 16 12:50:35.042024 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:35.041721 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" event={"ID":"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5","Type":"ContainerDied","Data":"00e04d9bba0b255123a55ba55ff18de42512f126d8c05dbd1dc3531033f8e7a3"} Sep 16 12:50:35.042024 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:35.042016 2584 scope.go:117] "RemoveContainer" containerID="00e04d9bba0b255123a55ba55ff18de42512f126d8c05dbd1dc3531033f8e7a3" Sep 16 12:50:36.048276 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:36.048235 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-vd2fc" event={"ID":"d0dc2888-aafe-47b0-bf94-0ba5fe2238c5","Type":"ContainerStarted","Data":"71f61749936c6bd0ca130f59d057e14cd1ce197bb266904414d20b2b2f944694"} Sep 16 12:50:39.007891 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:39.007853 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:50:39.073535 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:50:39.073501 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:05.405591 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:05.405492 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:09.807351 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:09.807320 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:10.266370 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:10.266345 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:14.824758 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.824720 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f67954898-rtjkd"] Sep 16 12:51:14.828319 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.828300 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:14.831020 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.830999 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 16 12:51:14.831173 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.831068 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 16 12:51:14.831332 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.831318 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lwvxt\"" Sep 16 12:51:14.831392 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.831341 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 16 12:51:14.831392 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.831358 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 16 12:51:14.831515 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.831476 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 16 12:51:14.836878 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.836857 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 16 12:51:14.842195 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.842152 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f67954898-rtjkd"] Sep 16 12:51:14.981482 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.981442 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:14.981482 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.981485 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-secret-telemeter-client\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:14.981695 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.981505 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:14.981695 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.981542 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-metrics-client-ca\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:14.981695 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.981569 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pz24\" (UniqueName: \"kubernetes.io/projected/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-kube-api-access-5pz24\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:14.981695 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.981588 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-telemeter-client-tls\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:14.981695 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.981616 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-federate-client-tls\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:14.981695 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:14.981636 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-serving-certs-ca-bundle\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.082373 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.082284 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-telemeter-client-tls\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.082373 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.082324 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-federate-client-tls\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.082373 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.082347 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-serving-certs-ca-bundle\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.082654 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.082393 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.082654 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.082416 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-secret-telemeter-client\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.082654 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.082444 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.082654 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.082474 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-metrics-client-ca\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.082654 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.082521 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pz24\" (UniqueName: \"kubernetes.io/projected/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-kube-api-access-5pz24\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.083317 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.083292 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-serving-certs-ca-bundle\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.083511 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.083333 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-metrics-client-ca\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.083643 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.083597 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.085055 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.085029 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-federate-client-tls\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.085517 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.085499 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-secret-telemeter-client\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.085639 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.085619 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.085681 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.085626 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-telemeter-client-tls\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.092712 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.092693 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pz24\" (UniqueName: \"kubernetes.io/projected/e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38-kube-api-access-5pz24\") pod \"telemeter-client-5f67954898-rtjkd\" (UID: \"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38\") " pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.138363 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.138337 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" Sep 16 12:51:15.418225 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:15.418130 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f67954898-rtjkd"] Sep 16 12:51:15.421031 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:51:15.421008 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71ebd8f_bb3f_4e29_9a06_8d4312c6eb38.slice/crio-3827571a4fd2a9ae81f6d43a04c80e1cc4108297cfe0aa55712477a07bf89301 WatchSource:0}: Error finding container 3827571a4fd2a9ae81f6d43a04c80e1cc4108297cfe0aa55712477a07bf89301: Status 404 returned error can't find the container with id 3827571a4fd2a9ae81f6d43a04c80e1cc4108297cfe0aa55712477a07bf89301 Sep 16 12:51:16.202599 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:16.202563 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" event={"ID":"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38","Type":"ContainerStarted","Data":"3827571a4fd2a9ae81f6d43a04c80e1cc4108297cfe0aa55712477a07bf89301"} Sep 16 12:51:18.212659 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:18.212620 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" event={"ID":"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38","Type":"ContainerStarted","Data":"e182cc044bb650c83f8dc9a1a2215b0628b687b41bdef860352c174b65714cb6"} Sep 16 12:51:18.213038 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:18.212675 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" event={"ID":"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38","Type":"ContainerStarted","Data":"069bdc4e7318ec0de1140cdd6a6b92f01fa10c4f2011ada26593d62e14a8c40e"} Sep 16 12:51:18.213038 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:18.212694 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" event={"ID":"e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38","Type":"ContainerStarted","Data":"d4ef9a88270c96641161683420032ffae8a6a88a13d7e2229a018941ebdda2f9"} Sep 16 12:51:18.237595 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:18.237552 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f67954898-rtjkd" podStartSLOduration=2.4108248469999998 podStartE2EDuration="4.237538737s" totalImagesPullingTime="1.82671389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:51:18.235854357 +0000 UTC m=+185.428502965" watchObservedRunningTime="2026-09-16 12:51:18.237538737 +0000 UTC m=+185.430187326" Sep 16 12:51:18.520079 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:18.520044 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:18.621245 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:18.621211 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:18.912850 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:18.912764 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:19.367351 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:19.367323 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:19.426592 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:19.426567 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:24.957784 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:24.957753 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:25.026740 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:25.026706 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:29.168593 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:29.168559 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:29.221999 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:29.221967 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:29.500622 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:29.500596 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:29.601661 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:29.601635 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:30.115412 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:30.115378 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:30.186422 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:30.186391 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:31.099382 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:31.099350 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:31.157107 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:31.157067 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:36.894564 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:36.894531 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:36.994994 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:36.994962 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:40.193063 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:40.193029 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:40.259111 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:40.259049 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:40.322103 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:40.322068 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:54.008222 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:54.008181 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:54.109207 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:54.109175 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:55.750462 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:55.750417 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:55.847207 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:55.847144 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:58.103524 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:58.103491 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:51:58.172301 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:51:58.172271 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:52:04.049651 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:04.049607 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:52:12.113393 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.113364 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:52:12.136116 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.136091 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h"] Sep 16 12:52:12.138219 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.138205 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" Sep 16 12:52:12.145979 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.145958 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zqtqc\"" Sep 16 12:52:12.146121 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.146101 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 12:52:12.146202 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.146113 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 16 12:52:12.146202 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.146190 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 12:52:12.146312 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.146103 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 12:52:12.148129 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.148105 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h"] Sep 16 12:52:12.214994 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.214967 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5eb0afb-4cee-4c4f-9eba-b4e282ab813d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h\" (UID: \"c5eb0afb-4cee-4c4f-9eba-b4e282ab813d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" Sep 16 12:52:12.215115 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.215024 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl9dn\" (UniqueName: \"kubernetes.io/projected/c5eb0afb-4cee-4c4f-9eba-b4e282ab813d-kube-api-access-zl9dn\") pod \"managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h\" (UID: \"c5eb0afb-4cee-4c4f-9eba-b4e282ab813d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" Sep 16 12:52:12.222152 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.222122 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:52:12.295170 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.295140 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:52:12.315867 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.315839 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl9dn\" (UniqueName: \"kubernetes.io/projected/c5eb0afb-4cee-4c4f-9eba-b4e282ab813d-kube-api-access-zl9dn\") pod \"managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h\" (UID: \"c5eb0afb-4cee-4c4f-9eba-b4e282ab813d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" Sep 16 12:52:12.315975 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.315884 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5eb0afb-4cee-4c4f-9eba-b4e282ab813d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h\" (UID: \"c5eb0afb-4cee-4c4f-9eba-b4e282ab813d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" Sep 16 12:52:12.318336 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.318316 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5eb0afb-4cee-4c4f-9eba-b4e282ab813d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h\" (UID: \"c5eb0afb-4cee-4c4f-9eba-b4e282ab813d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" Sep 16 12:52:12.324051 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.324030 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl9dn\" (UniqueName: \"kubernetes.io/projected/c5eb0afb-4cee-4c4f-9eba-b4e282ab813d-kube-api-access-zl9dn\") pod \"managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h\" (UID: \"c5eb0afb-4cee-4c4f-9eba-b4e282ab813d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" Sep 16 12:52:12.457770 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.457702 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" Sep 16 12:52:12.725534 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:12.725505 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h"] Sep 16 12:52:12.727525 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:52:12.727497 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5eb0afb_4cee_4c4f_9eba_b4e282ab813d.slice/crio-f1ee1f5a5b364824133bb0d3644b22bd4a9e3bbbf8a73919686ea73fe0f08500 WatchSource:0}: Error finding container f1ee1f5a5b364824133bb0d3644b22bd4a9e3bbbf8a73919686ea73fe0f08500: Status 404 returned error can't find the container with id f1ee1f5a5b364824133bb0d3644b22bd4a9e3bbbf8a73919686ea73fe0f08500 Sep 16 12:52:13.420491 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:13.420452 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" event={"ID":"c5eb0afb-4cee-4c4f-9eba-b4e282ab813d","Type":"ContainerStarted","Data":"f1ee1f5a5b364824133bb0d3644b22bd4a9e3bbbf8a73919686ea73fe0f08500"} Sep 16 12:52:15.429651 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:15.429612 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" event={"ID":"c5eb0afb-4cee-4c4f-9eba-b4e282ab813d","Type":"ContainerStarted","Data":"39f40766076fa6649280638cea194e8cf99ab07dbe311a72c39a6433bdb2df67"} Sep 16 12:52:15.447143 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:52:15.447095 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f46d6b8d6-zm46h" podStartSLOduration=0.887797066 podStartE2EDuration="3.447079325s" totalImagesPullingTime="2.559282259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:52:15.445210005 +0000 UTC m=+242.637858593" watchObservedRunningTime="2026-09-16 12:52:15.447079325 +0000 UTC m=+242.639727936" Sep 16 12:53:13.211384 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:53:13.211355 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 12:53:13.223014 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:53:13.222990 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 12:53:13.226680 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:53:13.226656 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 12:53:13.237515 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:53:13.237488 2584 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 12:53:13.240042 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:53:13.240021 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 12:53:39.637971 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:53:39.637917 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:53:55.657544 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:53:55.657502 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:53:55.741383 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:53:55.741338 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:53:57.492470 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:53:57.492435 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:53:57.563467 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:53:57.563438 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:54:33.686797 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:54:33.686758 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:54:33.941688 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:54:33.941571 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:54:47.192957 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:54:47.192925 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:54:57.935181 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:54:57.935131 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:54:58.033679 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:54:58.033647 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:54:58.099926 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:54:58.099894 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:11.371388 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:11.371354 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:11.445701 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:11.445670 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:14.414279 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:14.414245 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:14.652116 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:14.652078 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:14.738989 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:14.738964 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:15.737836 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.737799 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72"] Sep 16 12:57:15.740232 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.740216 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:15.744084 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.744061 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vwcvw\"" Sep 16 12:57:15.744226 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.744084 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 12:57:15.744226 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.744104 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 12:57:15.765938 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.765909 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72"] Sep 16 12:57:15.817689 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.817654 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:15.817862 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.817703 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl98f\" (UniqueName: \"kubernetes.io/projected/34e70df6-fade-43d2-8ff8-6ac16e99042f-kube-api-access-fl98f\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:15.817862 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.817795 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:15.918149 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.918115 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:15.918345 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.918184 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:15.918345 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.918226 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl98f\" (UniqueName: \"kubernetes.io/projected/34e70df6-fade-43d2-8ff8-6ac16e99042f-kube-api-access-fl98f\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:15.918499 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.918481 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:15.918545 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.918513 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:15.929173 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:15.929138 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl98f\" (UniqueName: \"kubernetes.io/projected/34e70df6-fade-43d2-8ff8-6ac16e99042f-kube-api-access-fl98f\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:16.049492 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:16.049393 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:16.325027 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:16.324948 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72"] Sep 16 12:57:16.330252 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:57:16.330225 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e70df6_fade_43d2_8ff8_6ac16e99042f.slice/crio-bfef33506a32f1b90093c43d83baae4c5cab2a13ee389494b5c3c20020241847 WatchSource:0}: Error finding container bfef33506a32f1b90093c43d83baae4c5cab2a13ee389494b5c3c20020241847: Status 404 returned error can't find the container with id bfef33506a32f1b90093c43d83baae4c5cab2a13ee389494b5c3c20020241847 Sep 16 12:57:16.332137 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:16.332118 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 12:57:16.547561 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:16.547530 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" event={"ID":"34e70df6-fade-43d2-8ff8-6ac16e99042f","Type":"ContainerStarted","Data":"bfef33506a32f1b90093c43d83baae4c5cab2a13ee389494b5c3c20020241847"} Sep 16 12:57:22.574031 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:22.573995 2584 generic.go:356] "Generic (PLEG): container finished" podID="34e70df6-fade-43d2-8ff8-6ac16e99042f" containerID="bce154c66389f907e97a2a432ea5ef5bf9f6d6d1362c463b5b0f646dccec051f" exitCode=0 Sep 16 12:57:22.574498 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:22.574082 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" event={"ID":"34e70df6-fade-43d2-8ff8-6ac16e99042f","Type":"ContainerDied","Data":"bce154c66389f907e97a2a432ea5ef5bf9f6d6d1362c463b5b0f646dccec051f"} Sep 16 12:57:24.584662 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:24.584625 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" event={"ID":"34e70df6-fade-43d2-8ff8-6ac16e99042f","Type":"ContainerStarted","Data":"713a81a8da9fb85e49d71c9b581e432143293383551f9dc72b32ae686536166a"} Sep 16 12:57:25.592723 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:25.592687 2584 generic.go:356] "Generic (PLEG): container finished" podID="34e70df6-fade-43d2-8ff8-6ac16e99042f" containerID="713a81a8da9fb85e49d71c9b581e432143293383551f9dc72b32ae686536166a" exitCode=0 Sep 16 12:57:25.593140 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:25.592733 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" event={"ID":"34e70df6-fade-43d2-8ff8-6ac16e99042f","Type":"ContainerDied","Data":"713a81a8da9fb85e49d71c9b581e432143293383551f9dc72b32ae686536166a"} Sep 16 12:57:32.623693 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:32.623662 2584 generic.go:356] "Generic (PLEG): container finished" podID="34e70df6-fade-43d2-8ff8-6ac16e99042f" containerID="f4a2e272fb75382cfcb6e821ef34ced9735006c442054c63140fb8ccc97a0bf1" exitCode=0 Sep 16 12:57:32.624073 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:32.623754 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" event={"ID":"34e70df6-fade-43d2-8ff8-6ac16e99042f","Type":"ContainerDied","Data":"f4a2e272fb75382cfcb6e821ef34ced9735006c442054c63140fb8ccc97a0bf1"} Sep 16 12:57:33.750510 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:33.750483 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:33.754124 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:33.754104 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl98f\" (UniqueName: \"kubernetes.io/projected/34e70df6-fade-43d2-8ff8-6ac16e99042f-kube-api-access-fl98f\") pod \"34e70df6-fade-43d2-8ff8-6ac16e99042f\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " Sep 16 12:57:33.754204 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:33.754136 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-bundle\") pod \"34e70df6-fade-43d2-8ff8-6ac16e99042f\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " Sep 16 12:57:33.754204 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:33.754177 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-util\") pod \"34e70df6-fade-43d2-8ff8-6ac16e99042f\" (UID: \"34e70df6-fade-43d2-8ff8-6ac16e99042f\") " Sep 16 12:57:33.754759 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:33.754731 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-bundle" (OuterVolumeSpecName: "bundle") pod "34e70df6-fade-43d2-8ff8-6ac16e99042f" (UID: "34e70df6-fade-43d2-8ff8-6ac16e99042f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 12:57:33.756330 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:33.756304 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e70df6-fade-43d2-8ff8-6ac16e99042f-kube-api-access-fl98f" (OuterVolumeSpecName: "kube-api-access-fl98f") pod "34e70df6-fade-43d2-8ff8-6ac16e99042f" (UID: "34e70df6-fade-43d2-8ff8-6ac16e99042f"). InnerVolumeSpecName "kube-api-access-fl98f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:57:33.759139 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:33.759107 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-util" (OuterVolumeSpecName: "util") pod "34e70df6-fade-43d2-8ff8-6ac16e99042f" (UID: "34e70df6-fade-43d2-8ff8-6ac16e99042f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 12:57:33.855335 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:33.855298 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl98f\" (UniqueName: \"kubernetes.io/projected/34e70df6-fade-43d2-8ff8-6ac16e99042f-kube-api-access-fl98f\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 12:57:33.855335 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:33.855326 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-bundle\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 12:57:33.855335 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:33.855338 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34e70df6-fade-43d2-8ff8-6ac16e99042f-util\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 12:57:34.633028 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:34.632992 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" event={"ID":"34e70df6-fade-43d2-8ff8-6ac16e99042f","Type":"ContainerDied","Data":"bfef33506a32f1b90093c43d83baae4c5cab2a13ee389494b5c3c20020241847"} Sep 16 12:57:34.633028 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:34.633035 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfef33506a32f1b90093c43d83baae4c5cab2a13ee389494b5c3c20020241847" Sep 16 12:57:34.633255 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:34.633090 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1annh72" Sep 16 12:57:37.962598 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:37.962533 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:38.065066 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:38.065027 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:39.799605 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.799572 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2"] Sep 16 12:57:39.799960 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.799871 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e70df6-fade-43d2-8ff8-6ac16e99042f" containerName="pull" Sep 16 12:57:39.799960 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.799884 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e70df6-fade-43d2-8ff8-6ac16e99042f" containerName="pull" Sep 16 12:57:39.799960 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.799903 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e70df6-fade-43d2-8ff8-6ac16e99042f" containerName="extract" Sep 16 12:57:39.799960 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.799909 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e70df6-fade-43d2-8ff8-6ac16e99042f" containerName="extract" Sep 16 12:57:39.799960 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.799918 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e70df6-fade-43d2-8ff8-6ac16e99042f" containerName="util" Sep 16 12:57:39.799960 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.799923 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e70df6-fade-43d2-8ff8-6ac16e99042f" containerName="util" Sep 16 12:57:39.800140 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.799974 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e70df6-fade-43d2-8ff8-6ac16e99042f" containerName="extract" Sep 16 12:57:39.802451 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.802435 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" Sep 16 12:57:39.806572 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.806549 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 16 12:57:39.806572 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.806565 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 16 12:57:39.806809 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.806793 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 16 12:57:39.806870 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.806821 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-cqmt5\"" Sep 16 12:57:39.815600 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.815580 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2"] Sep 16 12:57:39.892423 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.892394 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9a8f369d-53d2-46c8-a430-611994d1d1bc-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2nh2\" (UID: \"9a8f369d-53d2-46c8-a430-611994d1d1bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" Sep 16 12:57:39.892590 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.892467 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jk2g\" (UniqueName: \"kubernetes.io/projected/9a8f369d-53d2-46c8-a430-611994d1d1bc-kube-api-access-8jk2g\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2nh2\" (UID: \"9a8f369d-53d2-46c8-a430-611994d1d1bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" Sep 16 12:57:39.993258 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.993224 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9a8f369d-53d2-46c8-a430-611994d1d1bc-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2nh2\" (UID: \"9a8f369d-53d2-46c8-a430-611994d1d1bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" Sep 16 12:57:39.993431 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.993278 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jk2g\" (UniqueName: \"kubernetes.io/projected/9a8f369d-53d2-46c8-a430-611994d1d1bc-kube-api-access-8jk2g\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2nh2\" (UID: \"9a8f369d-53d2-46c8-a430-611994d1d1bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" Sep 16 12:57:39.995560 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:39.995531 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9a8f369d-53d2-46c8-a430-611994d1d1bc-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2nh2\" (UID: \"9a8f369d-53d2-46c8-a430-611994d1d1bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" Sep 16 12:57:40.004654 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:40.004630 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jk2g\" (UniqueName: \"kubernetes.io/projected/9a8f369d-53d2-46c8-a430-611994d1d1bc-kube-api-access-8jk2g\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-v2nh2\" (UID: \"9a8f369d-53d2-46c8-a430-611994d1d1bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" Sep 16 12:57:40.112666 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:40.112588 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" Sep 16 12:57:40.396555 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:40.396526 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2"] Sep 16 12:57:40.398798 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:57:40.398771 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8f369d_53d2_46c8_a430_611994d1d1bc.slice/crio-b5a0593572451a303cdca8a03466bed5fa2067a89af42b0cad8111ee67441f04 WatchSource:0}: Error finding container b5a0593572451a303cdca8a03466bed5fa2067a89af42b0cad8111ee67441f04: Status 404 returned error can't find the container with id b5a0593572451a303cdca8a03466bed5fa2067a89af42b0cad8111ee67441f04 Sep 16 12:57:40.657744 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:40.657666 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" event={"ID":"9a8f369d-53d2-46c8-a430-611994d1d1bc","Type":"ContainerStarted","Data":"b5a0593572451a303cdca8a03466bed5fa2067a89af42b0cad8111ee67441f04"} Sep 16 12:57:45.318293 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.318241 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:45.620375 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.620279 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:45.627914 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.627882 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-nhdpk"] Sep 16 12:57:45.631220 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.631196 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:45.631696 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.631674 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ff840f4-2d61-44e7-a03f-957befd8952d-certificates\") pod \"keda-operator-579c8958f7-nhdpk\" (UID: \"5ff840f4-2d61-44e7-a03f-957befd8952d\") " pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:45.631807 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.631761 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7666g\" (UniqueName: \"kubernetes.io/projected/5ff840f4-2d61-44e7-a03f-957befd8952d-kube-api-access-7666g\") pod \"keda-operator-579c8958f7-nhdpk\" (UID: \"5ff840f4-2d61-44e7-a03f-957befd8952d\") " pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:45.631884 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.631860 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5ff840f4-2d61-44e7-a03f-957befd8952d-cabundle0\") pod \"keda-operator-579c8958f7-nhdpk\" (UID: \"5ff840f4-2d61-44e7-a03f-957befd8952d\") " pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:45.634175 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.634146 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vmlnf\"" Sep 16 12:57:45.634389 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.634370 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 16 12:57:45.634613 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.634601 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Sep 16 12:57:45.644792 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.644770 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-nhdpk"] Sep 16 12:57:45.681757 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.681735 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" event={"ID":"9a8f369d-53d2-46c8-a430-611994d1d1bc","Type":"ContainerStarted","Data":"971f514cf11d1e8bcd358fc35a73a2c0b9ca3f144c22cd6c881a3c418b5bf6a3"} Sep 16 12:57:45.681871 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.681863 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" Sep 16 12:57:45.708009 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.707965 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" podStartSLOduration=2.264886256 podStartE2EDuration="6.707951524s" totalImagesPullingTime="4.443065268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:57:45.706846626 +0000 UTC m=+572.899495240" watchObservedRunningTime="2026-09-16 12:57:45.707951524 +0000 UTC m=+572.900600112" Sep 16 12:57:45.732483 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.732456 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5ff840f4-2d61-44e7-a03f-957befd8952d-cabundle0\") pod \"keda-operator-579c8958f7-nhdpk\" (UID: \"5ff840f4-2d61-44e7-a03f-957befd8952d\") " pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:45.732613 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.732503 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ff840f4-2d61-44e7-a03f-957befd8952d-certificates\") pod \"keda-operator-579c8958f7-nhdpk\" (UID: \"5ff840f4-2d61-44e7-a03f-957befd8952d\") " pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:45.732613 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.732568 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7666g\" (UniqueName: \"kubernetes.io/projected/5ff840f4-2d61-44e7-a03f-957befd8952d-kube-api-access-7666g\") pod \"keda-operator-579c8958f7-nhdpk\" (UID: \"5ff840f4-2d61-44e7-a03f-957befd8952d\") " pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:45.732743 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:57:45.732710 2584 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Sep 16 12:57:45.732803 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:57:45.732750 2584 secret.go:281] references non-existent secret key: ca.crt Sep 16 12:57:45.732803 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:57:45.732762 2584 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Sep 16 12:57:45.732803 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:57:45.732777 2584 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-nhdpk: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Sep 16 12:57:45.732970 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:57:45.732837 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ff840f4-2d61-44e7-a03f-957befd8952d-certificates podName:5ff840f4-2d61-44e7-a03f-957befd8952d nodeName:}" failed. No retries permitted until 2026-09-16 12:57:46.232818938 +0000 UTC m=+573.425467530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5ff840f4-2d61-44e7-a03f-957befd8952d-certificates") pod "keda-operator-579c8958f7-nhdpk" (UID: "5ff840f4-2d61-44e7-a03f-957befd8952d") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Sep 16 12:57:45.733230 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.733209 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5ff840f4-2d61-44e7-a03f-957befd8952d-cabundle0\") pod \"keda-operator-579c8958f7-nhdpk\" (UID: \"5ff840f4-2d61-44e7-a03f-957befd8952d\") " pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:45.747225 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.747196 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7666g\" (UniqueName: \"kubernetes.io/projected/5ff840f4-2d61-44e7-a03f-957befd8952d-kube-api-access-7666g\") pod \"keda-operator-579c8958f7-nhdpk\" (UID: \"5ff840f4-2d61-44e7-a03f-957befd8952d\") " pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:45.893151 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:45.893041 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:46.037172 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.037138 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:46.103444 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.103413 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn"] Sep 16 12:57:46.107100 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.107081 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.114010 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.113983 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Sep 16 12:57:46.120133 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.120107 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn"] Sep 16 12:57:46.134311 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.134276 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/905394ad-2732-48e0-8684-6ded35b15a76-certificates\") pod \"keda-metrics-apiserver-77857b7b98-js6jn\" (UID: \"905394ad-2732-48e0-8684-6ded35b15a76\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.134434 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.134312 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/905394ad-2732-48e0-8684-6ded35b15a76-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-js6jn\" (UID: \"905394ad-2732-48e0-8684-6ded35b15a76\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.134434 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.134337 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj46l\" (UniqueName: \"kubernetes.io/projected/905394ad-2732-48e0-8684-6ded35b15a76-kube-api-access-fj46l\") pod \"keda-metrics-apiserver-77857b7b98-js6jn\" (UID: \"905394ad-2732-48e0-8684-6ded35b15a76\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.234952 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.234925 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ff840f4-2d61-44e7-a03f-957befd8952d-certificates\") pod \"keda-operator-579c8958f7-nhdpk\" (UID: \"5ff840f4-2d61-44e7-a03f-957befd8952d\") " pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:46.235060 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.234957 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/905394ad-2732-48e0-8684-6ded35b15a76-certificates\") pod \"keda-metrics-apiserver-77857b7b98-js6jn\" (UID: \"905394ad-2732-48e0-8684-6ded35b15a76\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.235060 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.234989 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/905394ad-2732-48e0-8684-6ded35b15a76-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-js6jn\" (UID: \"905394ad-2732-48e0-8684-6ded35b15a76\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.235060 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.235013 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj46l\" (UniqueName: \"kubernetes.io/projected/905394ad-2732-48e0-8684-6ded35b15a76-kube-api-access-fj46l\") pod \"keda-metrics-apiserver-77857b7b98-js6jn\" (UID: \"905394ad-2732-48e0-8684-6ded35b15a76\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.235458 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.235439 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/905394ad-2732-48e0-8684-6ded35b15a76-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-js6jn\" (UID: \"905394ad-2732-48e0-8684-6ded35b15a76\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.237364 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.237344 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ff840f4-2d61-44e7-a03f-957befd8952d-certificates\") pod \"keda-operator-579c8958f7-nhdpk\" (UID: \"5ff840f4-2d61-44e7-a03f-957befd8952d\") " pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:46.237471 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.237453 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/905394ad-2732-48e0-8684-6ded35b15a76-certificates\") pod \"keda-metrics-apiserver-77857b7b98-js6jn\" (UID: \"905394ad-2732-48e0-8684-6ded35b15a76\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.240719 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.240690 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:46.243685 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.243661 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj46l\" (UniqueName: \"kubernetes.io/projected/905394ad-2732-48e0-8684-6ded35b15a76-kube-api-access-fj46l\") pod \"keda-metrics-apiserver-77857b7b98-js6jn\" (UID: \"905394ad-2732-48e0-8684-6ded35b15a76\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.425484 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.425455 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:46.466306 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.466262 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:46.478552 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.478521 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-kpb2x"] Sep 16 12:57:46.481874 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.481854 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:57:46.488059 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.487979 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Sep 16 12:57:46.496424 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.496336 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-kpb2x"] Sep 16 12:57:46.518223 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.517836 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-nhdpk"] Sep 16 12:57:46.520196 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:57:46.520132 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff840f4_2d61_44e7_a03f_957befd8952d.slice/crio-5e63cf88927efcc43ca4494a35b4508caee782b2891f4ffe234fb27b99be09d2 WatchSource:0}: Error finding container 5e63cf88927efcc43ca4494a35b4508caee782b2891f4ffe234fb27b99be09d2: Status 404 returned error can't find the container with id 5e63cf88927efcc43ca4494a35b4508caee782b2891f4ffe234fb27b99be09d2 Sep 16 12:57:46.537126 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.537103 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc9dg\" (UniqueName: \"kubernetes.io/projected/652a9cb8-34b1-441d-9939-7794f196c439-kube-api-access-bc9dg\") pod \"keda-admission-6c4d864b86-kpb2x\" (UID: \"652a9cb8-34b1-441d-9939-7794f196c439\") " pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:57:46.537247 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.537209 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/652a9cb8-34b1-441d-9939-7794f196c439-certificates\") pod \"keda-admission-6c4d864b86-kpb2x\" (UID: \"652a9cb8-34b1-441d-9939-7794f196c439\") " pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:57:46.567530 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.567503 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:46.638039 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.638008 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/652a9cb8-34b1-441d-9939-7794f196c439-certificates\") pod \"keda-admission-6c4d864b86-kpb2x\" (UID: \"652a9cb8-34b1-441d-9939-7794f196c439\") " pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:57:46.638243 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.638101 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9dg\" (UniqueName: \"kubernetes.io/projected/652a9cb8-34b1-441d-9939-7794f196c439-kube-api-access-bc9dg\") pod \"keda-admission-6c4d864b86-kpb2x\" (UID: \"652a9cb8-34b1-441d-9939-7794f196c439\") " pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:57:46.638243 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:57:46.638182 2584 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Sep 16 12:57:46.638243 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:57:46.638207 2584 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-kpb2x: secret "keda-admission-webhooks-certs" not found Sep 16 12:57:46.638401 ip-10-0-141-139 kubenswrapper[2584]: E0916 12:57:46.638260 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/652a9cb8-34b1-441d-9939-7794f196c439-certificates podName:652a9cb8-34b1-441d-9939-7794f196c439 nodeName:}" failed. No retries permitted until 2026-09-16 12:57:47.138243031 +0000 UTC m=+574.330891600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/652a9cb8-34b1-441d-9939-7794f196c439-certificates") pod "keda-admission-6c4d864b86-kpb2x" (UID: "652a9cb8-34b1-441d-9939-7794f196c439") : secret "keda-admission-webhooks-certs" not found Sep 16 12:57:46.652866 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.652845 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9dg\" (UniqueName: \"kubernetes.io/projected/652a9cb8-34b1-441d-9939-7794f196c439-kube-api-access-bc9dg\") pod \"keda-admission-6c4d864b86-kpb2x\" (UID: \"652a9cb8-34b1-441d-9939-7794f196c439\") " pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:57:46.686976 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.686947 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-nhdpk" event={"ID":"5ff840f4-2d61-44e7-a03f-957befd8952d","Type":"ContainerStarted","Data":"5e63cf88927efcc43ca4494a35b4508caee782b2891f4ffe234fb27b99be09d2"} Sep 16 12:57:46.711220 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.711193 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn"] Sep 16 12:57:46.712063 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:57:46.712039 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905394ad_2732_48e0_8684_6ded35b15a76.slice/crio-039fb0243495176785e7b03de5f40334f3da40e70459def001108050dbb88f3c WatchSource:0}: Error finding container 039fb0243495176785e7b03de5f40334f3da40e70459def001108050dbb88f3c: Status 404 returned error can't find the container with id 039fb0243495176785e7b03de5f40334f3da40e70459def001108050dbb88f3c Sep 16 12:57:46.860781 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:46.860693 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:47.057611 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:47.057570 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:47.142332 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:47.142256 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/652a9cb8-34b1-441d-9939-7794f196c439-certificates\") pod \"keda-admission-6c4d864b86-kpb2x\" (UID: \"652a9cb8-34b1-441d-9939-7794f196c439\") " pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:57:47.144694 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:47.144677 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/652a9cb8-34b1-441d-9939-7794f196c439-certificates\") pod \"keda-admission-6c4d864b86-kpb2x\" (UID: \"652a9cb8-34b1-441d-9939-7794f196c439\") " pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:57:47.258137 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:47.258090 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:47.354591 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:47.354557 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:47.399328 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:47.399256 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:57:47.454053 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:47.453730 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:57:47.692050 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:47.691975 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-kpb2x"] Sep 16 12:57:47.696283 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:47.696248 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" event={"ID":"905394ad-2732-48e0-8684-6ded35b15a76","Type":"ContainerStarted","Data":"039fb0243495176785e7b03de5f40334f3da40e70459def001108050dbb88f3c"} Sep 16 12:57:47.696405 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:57:47.696318 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652a9cb8_34b1_441d_9939_7794f196c439.slice/crio-b096e60e1ffd6ac51472ffd85a126563ac62b7c20ad6641ad1701c4e5ce9bc9d WatchSource:0}: Error finding container b096e60e1ffd6ac51472ffd85a126563ac62b7c20ad6641ad1701c4e5ce9bc9d: Status 404 returned error can't find the container with id b096e60e1ffd6ac51472ffd85a126563ac62b7c20ad6641ad1701c4e5ce9bc9d Sep 16 12:57:48.704188 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:48.704135 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" event={"ID":"652a9cb8-34b1-441d-9939-7794f196c439","Type":"ContainerStarted","Data":"b096e60e1ffd6ac51472ffd85a126563ac62b7c20ad6641ad1701c4e5ce9bc9d"} Sep 16 12:57:51.720658 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:51.720605 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-nhdpk" event={"ID":"5ff840f4-2d61-44e7-a03f-957befd8952d","Type":"ContainerStarted","Data":"40c96f974b218d1fe7ad6c2a499929df18a10518be2f2ad4bee4fa11e2e3962b"} Sep 16 12:57:51.721106 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:51.720825 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:57:51.722352 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:51.722329 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" event={"ID":"905394ad-2732-48e0-8684-6ded35b15a76","Type":"ContainerStarted","Data":"cb17184e6b13b3a3d31a25abb842b7d3dadada363251362be5d8654550e1f7fb"} Sep 16 12:57:51.722469 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:51.722433 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:57:51.724016 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:51.723993 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" event={"ID":"652a9cb8-34b1-441d-9939-7794f196c439","Type":"ContainerStarted","Data":"adedab88e0e96c35f8d69f5541b5c3fcc1b77d1013491523e4ae3f17c6ab2309"} Sep 16 12:57:51.724099 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:51.724046 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:57:51.738583 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:51.738540 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-nhdpk" podStartSLOduration=2.369076685 podStartE2EDuration="6.738526598s" totalImagesPullingTime="4.369449913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:57:51.738127435 +0000 UTC m=+578.930776049" watchObservedRunningTime="2026-09-16 12:57:51.738526598 +0000 UTC m=+578.931175209" Sep 16 12:57:51.756652 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:51.756611 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" podStartSLOduration=1.58892531 podStartE2EDuration="5.756598079s" totalImagesPullingTime="4.167672769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:57:51.755790548 +0000 UTC m=+578.948439157" watchObservedRunningTime="2026-09-16 12:57:51.756598079 +0000 UTC m=+578.949246669" Sep 16 12:57:51.778256 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:57:51.774528 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" podStartSLOduration=2.593629891 podStartE2EDuration="5.774499727s" totalImagesPullingTime="3.180869836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:57:51.773080773 +0000 UTC m=+578.965729372" watchObservedRunningTime="2026-09-16 12:57:51.774499727 +0000 UTC m=+578.967148317" Sep 16 12:58:02.732906 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:02.732874 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-js6jn" Sep 16 12:58:06.690259 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:06.690224 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-v2nh2" Sep 16 12:58:12.730984 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:12.730952 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-nhdpk" Sep 16 12:58:12.731417 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:12.731293 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-kpb2x" Sep 16 12:58:13.259684 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:13.259653 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 12:58:13.260543 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:13.260521 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 12:58:13.268262 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:13.268238 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 12:58:13.269503 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:13.269470 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 12:58:42.200748 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:42.200714 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:58:44.089610 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:44.089578 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:58:59.809952 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:58:59.809921 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:59:07.047113 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:07.047084 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:59:13.789683 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.789648 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:59:13.891404 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.891370 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-867644d759-zm4tp"] Sep 16 12:59:13.894208 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.894185 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:13.897457 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.897436 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Sep 16 12:59:13.897572 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.897473 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 16 12:59:13.898789 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.898763 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 16 12:59:13.898887 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.898796 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-z8wn7\"" Sep 16 12:59:13.914767 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.914735 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-zm4tp"] Sep 16 12:59:13.936791 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.936765 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:59:13.942885 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.942859 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nlxx\" (UniqueName: \"kubernetes.io/projected/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-kube-api-access-9nlxx\") pod \"kserve-controller-manager-867644d759-zm4tp\" (UID: \"f8154241-f809-4d6c-ab94-0dc6f4c21bbc\") " pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:13.943013 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.942897 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-cert\") pod \"kserve-controller-manager-867644d759-zm4tp\" (UID: \"f8154241-f809-4d6c-ab94-0dc6f4c21bbc\") " pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:13.958990 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.958963 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-rj5f2"] Sep 16 12:59:13.961493 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.961474 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rj5f2" Sep 16 12:59:13.965337 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.965315 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dt9dl\"" Sep 16 12:59:13.965987 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.965973 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 16 12:59:13.979187 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:13.979146 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rj5f2"] Sep 16 12:59:14.043846 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.043779 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/533b15c8-d8dd-4b33-bb95-769fae375156-data\") pod \"seaweedfs-64568bcd49-rj5f2\" (UID: \"533b15c8-d8dd-4b33-bb95-769fae375156\") " pod="kserve/seaweedfs-64568bcd49-rj5f2" Sep 16 12:59:14.043846 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.043819 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nlxx\" (UniqueName: \"kubernetes.io/projected/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-kube-api-access-9nlxx\") pod \"kserve-controller-manager-867644d759-zm4tp\" (UID: \"f8154241-f809-4d6c-ab94-0dc6f4c21bbc\") " pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:14.043846 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.043847 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-cert\") pod \"kserve-controller-manager-867644d759-zm4tp\" (UID: \"f8154241-f809-4d6c-ab94-0dc6f4c21bbc\") " pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:14.044049 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.043898 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh9mc\" (UniqueName: \"kubernetes.io/projected/533b15c8-d8dd-4b33-bb95-769fae375156-kube-api-access-wh9mc\") pod \"seaweedfs-64568bcd49-rj5f2\" (UID: \"533b15c8-d8dd-4b33-bb95-769fae375156\") " pod="kserve/seaweedfs-64568bcd49-rj5f2" Sep 16 12:59:14.046454 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.046432 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-cert\") pod \"kserve-controller-manager-867644d759-zm4tp\" (UID: \"f8154241-f809-4d6c-ab94-0dc6f4c21bbc\") " pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:14.054862 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.054845 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nlxx\" (UniqueName: \"kubernetes.io/projected/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-kube-api-access-9nlxx\") pod \"kserve-controller-manager-867644d759-zm4tp\" (UID: \"f8154241-f809-4d6c-ab94-0dc6f4c21bbc\") " pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:14.077792 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.077769 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:59:14.145226 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.145197 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh9mc\" (UniqueName: \"kubernetes.io/projected/533b15c8-d8dd-4b33-bb95-769fae375156-kube-api-access-wh9mc\") pod \"seaweedfs-64568bcd49-rj5f2\" (UID: \"533b15c8-d8dd-4b33-bb95-769fae375156\") " pod="kserve/seaweedfs-64568bcd49-rj5f2" Sep 16 12:59:14.145375 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.145250 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/533b15c8-d8dd-4b33-bb95-769fae375156-data\") pod \"seaweedfs-64568bcd49-rj5f2\" (UID: \"533b15c8-d8dd-4b33-bb95-769fae375156\") " pod="kserve/seaweedfs-64568bcd49-rj5f2" Sep 16 12:59:14.145611 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.145591 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/533b15c8-d8dd-4b33-bb95-769fae375156-data\") pod \"seaweedfs-64568bcd49-rj5f2\" (UID: \"533b15c8-d8dd-4b33-bb95-769fae375156\") " pod="kserve/seaweedfs-64568bcd49-rj5f2" Sep 16 12:59:14.159711 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.159688 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh9mc\" (UniqueName: \"kubernetes.io/projected/533b15c8-d8dd-4b33-bb95-769fae375156-kube-api-access-wh9mc\") pod \"seaweedfs-64568bcd49-rj5f2\" (UID: \"533b15c8-d8dd-4b33-bb95-769fae375156\") " pod="kserve/seaweedfs-64568bcd49-rj5f2" Sep 16 12:59:14.204266 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.204242 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:14.256084 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.256024 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:59:14.271141 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.271116 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rj5f2" Sep 16 12:59:14.481813 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.481779 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-zm4tp"] Sep 16 12:59:14.489917 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:59:14.489894 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8154241_f809_4d6c_ab94_0dc6f4c21bbc.slice/crio-87c741c8b7974cb9e8da3e716f63f110cb2eb3703963e5dd6902b7fe4477b8e9 WatchSource:0}: Error finding container 87c741c8b7974cb9e8da3e716f63f110cb2eb3703963e5dd6902b7fe4477b8e9: Status 404 returned error can't find the container with id 87c741c8b7974cb9e8da3e716f63f110cb2eb3703963e5dd6902b7fe4477b8e9 Sep 16 12:59:14.547087 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.547066 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rj5f2"] Sep 16 12:59:14.548985 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:59:14.548961 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533b15c8_d8dd_4b33_bb95_769fae375156.slice/crio-f50662935d4c7f01e9ffd8b7dd373d6956c818b2d8a75f5c6373b32e616ba1a5 WatchSource:0}: Error finding container f50662935d4c7f01e9ffd8b7dd373d6956c818b2d8a75f5c6373b32e616ba1a5: Status 404 returned error can't find the container with id f50662935d4c7f01e9ffd8b7dd373d6956c818b2d8a75f5c6373b32e616ba1a5 Sep 16 12:59:14.782855 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.782820 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:59:14.875806 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.875769 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:59:14.953031 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:14.953000 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 12:59:15.099549 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:15.099473 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-zm4tp" event={"ID":"f8154241-f809-4d6c-ab94-0dc6f4c21bbc","Type":"ContainerStarted","Data":"87c741c8b7974cb9e8da3e716f63f110cb2eb3703963e5dd6902b7fe4477b8e9"} Sep 16 12:59:15.101312 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:15.101283 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rj5f2" event={"ID":"533b15c8-d8dd-4b33-bb95-769fae375156","Type":"ContainerStarted","Data":"f50662935d4c7f01e9ffd8b7dd373d6956c818b2d8a75f5c6373b32e616ba1a5"} Sep 16 12:59:19.125217 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:19.125181 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rj5f2" event={"ID":"533b15c8-d8dd-4b33-bb95-769fae375156","Type":"ContainerStarted","Data":"c85253fd64c5914d82843f29b13eae0467d3484084693a76bf9d641a3b751366"} Sep 16 12:59:19.125683 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:19.125333 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-rj5f2" Sep 16 12:59:19.126686 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:19.126660 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-zm4tp" event={"ID":"f8154241-f809-4d6c-ab94-0dc6f4c21bbc","Type":"ContainerStarted","Data":"bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7"} Sep 16 12:59:19.126788 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:19.126719 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:19.150453 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:19.150408 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-rj5f2" podStartSLOduration=2.514579458 podStartE2EDuration="6.15039418s" totalImagesPullingTime="3.635814722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:59:19.148685493 +0000 UTC m=+666.341334106" watchObservedRunningTime="2026-09-16 12:59:19.15039418 +0000 UTC m=+666.343042769" Sep 16 12:59:19.172120 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:19.172080 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-867644d759-zm4tp" podStartSLOduration=2.544182171 podStartE2EDuration="6.172066348s" totalImagesPullingTime="3.627884177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:59:19.170893041 +0000 UTC m=+666.363541656" watchObservedRunningTime="2026-09-16 12:59:19.172066348 +0000 UTC m=+666.364714936" Sep 16 12:59:25.132997 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:25.132962 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-rj5f2" Sep 16 12:59:50.135547 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:50.135514 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:59.191580 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.191545 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-867644d759-zm4tp"] Sep 16 12:59:59.192101 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.191764 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-867644d759-zm4tp" podUID="f8154241-f809-4d6c-ab94-0dc6f4c21bbc" containerName="manager" containerID="cri-o://bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7" gracePeriod=10 Sep 16 12:59:59.238976 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.238949 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-867644d759-8dqsv"] Sep 16 12:59:59.241371 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.241356 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-8dqsv" Sep 16 12:59:59.272885 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.272862 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-8dqsv"] Sep 16 12:59:59.303769 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.303742 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dfcabbe-dc77-4541-93d9-8ec5b8d132fc-cert\") pod \"kserve-controller-manager-867644d759-8dqsv\" (UID: \"5dfcabbe-dc77-4541-93d9-8ec5b8d132fc\") " pod="kserve/kserve-controller-manager-867644d759-8dqsv" Sep 16 12:59:59.303852 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.303790 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tzg\" (UniqueName: \"kubernetes.io/projected/5dfcabbe-dc77-4541-93d9-8ec5b8d132fc-kube-api-access-t7tzg\") pod \"kserve-controller-manager-867644d759-8dqsv\" (UID: \"5dfcabbe-dc77-4541-93d9-8ec5b8d132fc\") " pod="kserve/kserve-controller-manager-867644d759-8dqsv" Sep 16 12:59:59.404325 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.404295 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dfcabbe-dc77-4541-93d9-8ec5b8d132fc-cert\") pod \"kserve-controller-manager-867644d759-8dqsv\" (UID: \"5dfcabbe-dc77-4541-93d9-8ec5b8d132fc\") " pod="kserve/kserve-controller-manager-867644d759-8dqsv" Sep 16 12:59:59.404455 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.404345 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tzg\" (UniqueName: \"kubernetes.io/projected/5dfcabbe-dc77-4541-93d9-8ec5b8d132fc-kube-api-access-t7tzg\") pod \"kserve-controller-manager-867644d759-8dqsv\" (UID: \"5dfcabbe-dc77-4541-93d9-8ec5b8d132fc\") " pod="kserve/kserve-controller-manager-867644d759-8dqsv" Sep 16 12:59:59.406432 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.406411 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dfcabbe-dc77-4541-93d9-8ec5b8d132fc-cert\") pod \"kserve-controller-manager-867644d759-8dqsv\" (UID: \"5dfcabbe-dc77-4541-93d9-8ec5b8d132fc\") " pod="kserve/kserve-controller-manager-867644d759-8dqsv" Sep 16 12:59:59.413546 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.413523 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tzg\" (UniqueName: \"kubernetes.io/projected/5dfcabbe-dc77-4541-93d9-8ec5b8d132fc-kube-api-access-t7tzg\") pod \"kserve-controller-manager-867644d759-8dqsv\" (UID: \"5dfcabbe-dc77-4541-93d9-8ec5b8d132fc\") " pod="kserve/kserve-controller-manager-867644d759-8dqsv" Sep 16 12:59:59.428075 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.428054 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 12:59:59.568858 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.568824 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-8dqsv" Sep 16 12:59:59.606524 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.606488 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nlxx\" (UniqueName: \"kubernetes.io/projected/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-kube-api-access-9nlxx\") pod \"f8154241-f809-4d6c-ab94-0dc6f4c21bbc\" (UID: \"f8154241-f809-4d6c-ab94-0dc6f4c21bbc\") " Sep 16 12:59:59.606524 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.606533 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-cert\") pod \"f8154241-f809-4d6c-ab94-0dc6f4c21bbc\" (UID: \"f8154241-f809-4d6c-ab94-0dc6f4c21bbc\") " Sep 16 12:59:59.608673 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.608646 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-cert" (OuterVolumeSpecName: "cert") pod "f8154241-f809-4d6c-ab94-0dc6f4c21bbc" (UID: "f8154241-f809-4d6c-ab94-0dc6f4c21bbc"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:59:59.608673 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.608654 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-kube-api-access-9nlxx" (OuterVolumeSpecName: "kube-api-access-9nlxx") pod "f8154241-f809-4d6c-ab94-0dc6f4c21bbc" (UID: "f8154241-f809-4d6c-ab94-0dc6f4c21bbc"). InnerVolumeSpecName "kube-api-access-9nlxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:59:59.707106 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.707072 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nlxx\" (UniqueName: \"kubernetes.io/projected/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-kube-api-access-9nlxx\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 12:59:59.707106 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.707101 2584 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8154241-f809-4d6c-ab94-0dc6f4c21bbc-cert\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 12:59:59.842622 ip-10-0-141-139 kubenswrapper[2584]: I0916 12:59:59.842549 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-8dqsv"] Sep 16 12:59:59.845966 ip-10-0-141-139 kubenswrapper[2584]: W0916 12:59:59.845941 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfcabbe_dc77_4541_93d9_8ec5b8d132fc.slice/crio-6a109f86d30fc94a97e44136616804feda52b5e4a0492f6ca83f6b266e9c04aa WatchSource:0}: Error finding container 6a109f86d30fc94a97e44136616804feda52b5e4a0492f6ca83f6b266e9c04aa: Status 404 returned error can't find the container with id 6a109f86d30fc94a97e44136616804feda52b5e4a0492f6ca83f6b266e9c04aa Sep 16 13:00:00.308464 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:00.308431 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-8dqsv" event={"ID":"5dfcabbe-dc77-4541-93d9-8ec5b8d132fc","Type":"ContainerStarted","Data":"6a109f86d30fc94a97e44136616804feda52b5e4a0492f6ca83f6b266e9c04aa"} Sep 16 13:00:00.309640 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:00.309617 2584 generic.go:356] "Generic (PLEG): container finished" podID="f8154241-f809-4d6c-ab94-0dc6f4c21bbc" containerID="bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7" exitCode=0 Sep 16 13:00:00.309763 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:00.309670 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-zm4tp" Sep 16 13:00:00.309763 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:00.309677 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-zm4tp" event={"ID":"f8154241-f809-4d6c-ab94-0dc6f4c21bbc","Type":"ContainerDied","Data":"bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7"} Sep 16 13:00:00.309865 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:00.309784 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-zm4tp" event={"ID":"f8154241-f809-4d6c-ab94-0dc6f4c21bbc","Type":"ContainerDied","Data":"87c741c8b7974cb9e8da3e716f63f110cb2eb3703963e5dd6902b7fe4477b8e9"} Sep 16 13:00:00.309865 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:00.309806 2584 scope.go:117] "RemoveContainer" containerID="bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7" Sep 16 13:00:00.318451 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:00.318432 2584 scope.go:117] "RemoveContainer" containerID="bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7" Sep 16 13:00:00.318729 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:00:00.318705 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7\": container with ID starting with bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7 not found: ID does not exist" containerID="bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7" Sep 16 13:00:00.318819 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:00.318733 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7"} err="failed to get container status \"bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7\": rpc error: code = NotFound desc = could not find container \"bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7\": container with ID starting with bd7eaecb9c0d18f0c78926b3d38a8d23217a8b0f12dc6db04b66176adafc7aa7 not found: ID does not exist" Sep 16 13:00:00.332988 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:00.332968 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-867644d759-zm4tp"] Sep 16 13:00:00.335795 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:00.335774 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-867644d759-zm4tp"] Sep 16 13:00:01.317506 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:01.317467 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-8dqsv" event={"ID":"5dfcabbe-dc77-4541-93d9-8ec5b8d132fc","Type":"ContainerStarted","Data":"ca31ef82d4ceb2350b0e0b9e69dbd6ebad531b14ff0d9bebba26b519aa4decd9"} Sep 16 13:00:01.317865 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:01.317601 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-867644d759-8dqsv" Sep 16 13:00:01.342348 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:01.342299 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-867644d759-8dqsv" podStartSLOduration=1.839243511 podStartE2EDuration="2.342286095s" totalImagesPullingTime="503.042584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:01.342179977 +0000 UTC m=+708.534828577" watchObservedRunningTime="2026-09-16 13:00:01.342286095 +0000 UTC m=+708.534934683" Sep 16 13:00:01.472760 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:01.472728 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8154241-f809-4d6c-ab94-0dc6f4c21bbc" path="/var/lib/kubelet/pods/f8154241-f809-4d6c-ab94-0dc6f4c21bbc/volumes" Sep 16 13:00:32.328680 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:32.328649 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-867644d759-8dqsv" Sep 16 13:00:33.804065 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:33.804031 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:00:33.945447 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:33.945414 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:00:34.032945 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:34.032915 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:00:34.133039 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:34.132972 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:00:35.713893 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:35.713865 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:00:35.810606 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:00:35.810567 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:09.145382 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:09.145343 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:09.227997 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:09.227954 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:21.147203 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.147150 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:21.330930 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.330898 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:21.374054 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.374021 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8"] Sep 16 13:02:21.374413 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.374400 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8154241-f809-4d6c-ab94-0dc6f4c21bbc" containerName="manager" Sep 16 13:02:21.374459 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.374414 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8154241-f809-4d6c-ab94-0dc6f4c21bbc" containerName="manager" Sep 16 13:02:21.374493 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.374460 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8154241-f809-4d6c-ab94-0dc6f4c21bbc" containerName="manager" Sep 16 13:02:21.377467 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.377444 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:21.380115 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.380088 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-18839-predictor-serving-cert\"" Sep 16 13:02:21.380249 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.380182 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 13:02:21.380311 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.380259 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gdsnl\"" Sep 16 13:02:21.381417 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.381389 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-18839-kube-rbac-proxy-sar-config\"" Sep 16 13:02:21.381534 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.381446 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 13:02:21.387201 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.387180 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8"] Sep 16 13:02:21.419724 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.419642 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0633927-e158-43da-a712-ee87d0caa672-proxy-tls\") pod \"success-200-isvc-18839-predictor-7978f4cd9c-nchb8\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:21.419724 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.419696 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0633927-e158-43da-a712-ee87d0caa672-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-18839-predictor-7978f4cd9c-nchb8\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:21.419908 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.419731 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59nx\" (UniqueName: \"kubernetes.io/projected/d0633927-e158-43da-a712-ee87d0caa672-kube-api-access-f59nx\") pod \"success-200-isvc-18839-predictor-7978f4cd9c-nchb8\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:21.442271 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.442245 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:21.520323 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.520288 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0633927-e158-43da-a712-ee87d0caa672-proxy-tls\") pod \"success-200-isvc-18839-predictor-7978f4cd9c-nchb8\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:21.520545 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.520529 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0633927-e158-43da-a712-ee87d0caa672-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-18839-predictor-7978f4cd9c-nchb8\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:21.520684 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.520658 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59nx\" (UniqueName: \"kubernetes.io/projected/d0633927-e158-43da-a712-ee87d0caa672-kube-api-access-f59nx\") pod \"success-200-isvc-18839-predictor-7978f4cd9c-nchb8\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:21.520902 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:02:21.520416 2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-18839-predictor-serving-cert: secret "success-200-isvc-18839-predictor-serving-cert" not found Sep 16 13:02:21.521034 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:02:21.521024 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0633927-e158-43da-a712-ee87d0caa672-proxy-tls podName:d0633927-e158-43da-a712-ee87d0caa672 nodeName:}" failed. No retries permitted until 2026-09-16 13:02:22.021001884 +0000 UTC m=+849.213650468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d0633927-e158-43da-a712-ee87d0caa672-proxy-tls") pod "success-200-isvc-18839-predictor-7978f4cd9c-nchb8" (UID: "d0633927-e158-43da-a712-ee87d0caa672") : secret "success-200-isvc-18839-predictor-serving-cert" not found Sep 16 13:02:21.521316 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.521294 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0633927-e158-43da-a712-ee87d0caa672-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-18839-predictor-7978f4cd9c-nchb8\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:21.531885 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.531861 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59nx\" (UniqueName: \"kubernetes.io/projected/d0633927-e158-43da-a712-ee87d0caa672-kube-api-access-f59nx\") pod \"success-200-isvc-18839-predictor-7978f4cd9c-nchb8\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:21.544927 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.544888 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:21.640397 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.640368 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:21.714351 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.714265 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:21.772115 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:21.772083 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:22.026208 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:22.026144 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0633927-e158-43da-a712-ee87d0caa672-proxy-tls\") pod \"success-200-isvc-18839-predictor-7978f4cd9c-nchb8\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:22.028804 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:22.028764 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0633927-e158-43da-a712-ee87d0caa672-proxy-tls\") pod \"success-200-isvc-18839-predictor-7978f4cd9c-nchb8\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:22.076110 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:22.076082 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:02:22.289004 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:22.288930 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:22.559210 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:22.559185 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8"] Sep 16 13:02:22.561236 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:02:22.561214 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0633927_e158_43da_a712_ee87d0caa672.slice/crio-ff2e68470591a01deac7e42166ba2f61e8c0ec49465a8b383d9596ddff4e6a73 WatchSource:0}: Error finding container ff2e68470591a01deac7e42166ba2f61e8c0ec49465a8b383d9596ddff4e6a73: Status 404 returned error can't find the container with id ff2e68470591a01deac7e42166ba2f61e8c0ec49465a8b383d9596ddff4e6a73 Sep 16 13:02:22.563048 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:22.563032 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:02:22.938035 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:22.937961 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" event={"ID":"d0633927-e158-43da-a712-ee87d0caa672","Type":"ContainerStarted","Data":"ff2e68470591a01deac7e42166ba2f61e8c0ec49465a8b383d9596ddff4e6a73"} Sep 16 13:02:34.000361 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:34.000328 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" event={"ID":"d0633927-e158-43da-a712-ee87d0caa672","Type":"ContainerStarted","Data":"ba31021f2cc699b32bcdfdd97b35f51a1f64fc040eb10b8cef4ec5e81c60343a"} Sep 16 13:02:37.019403 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:37.019360 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" event={"ID":"d0633927-e158-43da-a712-ee87d0caa672","Type":"ContainerStarted","Data":"b73756b9422f0b47cdbbf0266393bc08a8b0603bd98540c300dc17253b535828"} Sep 16 13:02:37.019870 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:37.019434 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:37.019870 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:37.019488 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:37.020949 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:37.020912 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Sep 16 13:02:37.043068 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:37.043018 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podStartSLOduration=2.436685166 podStartE2EDuration="16.042999177s" totalImagesPullingTime="13.606314011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:37.040009951 +0000 UTC m=+864.232658540" watchObservedRunningTime="2026-09-16 13:02:37.042999177 +0000 UTC m=+864.235647766" Sep 16 13:02:38.024257 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:38.024214 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Sep 16 13:02:43.028068 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:43.028037 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:02:43.028546 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:43.028361 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Sep 16 13:02:53.028609 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:02:53.028563 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Sep 16 13:03:03.028780 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:03.028735 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Sep 16 13:03:13.028430 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:13.028392 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Sep 16 13:03:13.175124 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:13.175092 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:03:13.299147 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:13.299066 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:03:13.300109 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:13.300083 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:03:13.308636 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:13.308618 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:03:13.308636 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:13.308634 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:03:23.030125 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:23.030095 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:03:41.183057 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:41.183021 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:03:41.365368 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:41.365319 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:03:51.310008 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.309975 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:03:51.372782 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.372749 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8"] Sep 16 13:03:51.373012 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.372982 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" containerID="cri-o://ba31021f2cc699b32bcdfdd97b35f51a1f64fc040eb10b8cef4ec5e81c60343a" gracePeriod=30 Sep 16 13:03:51.373095 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.373017 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kube-rbac-proxy" containerID="cri-o://b73756b9422f0b47cdbbf0266393bc08a8b0603bd98540c300dc17253b535828" gracePeriod=30 Sep 16 13:03:51.487345 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.487314 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj"] Sep 16 13:03:51.490075 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.490057 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:51.492586 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.492565 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-03a3a-kube-rbac-proxy-sar-config\"" Sep 16 13:03:51.492710 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.492589 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-03a3a-predictor-serving-cert\"" Sep 16 13:03:51.499805 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.499782 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj"] Sep 16 13:03:51.580236 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.580150 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:51.580394 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.580235 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsx6c\" (UniqueName: \"kubernetes.io/projected/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-api-access-rsx6c\") pod \"success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:51.580394 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.580276 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d516bad0-cbd0-4c98-8dd1-729170b4d151-proxy-tls\") pod \"success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:51.681356 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.681318 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsx6c\" (UniqueName: \"kubernetes.io/projected/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-api-access-rsx6c\") pod \"success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:51.681523 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.681368 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d516bad0-cbd0-4c98-8dd1-729170b4d151-proxy-tls\") pod \"success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:51.681523 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.681416 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:51.681639 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:03:51.681533 2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-serving-cert: secret "success-200-isvc-03a3a-predictor-serving-cert" not found Sep 16 13:03:51.681639 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:03:51.681617 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d516bad0-cbd0-4c98-8dd1-729170b4d151-proxy-tls podName:d516bad0-cbd0-4c98-8dd1-729170b4d151 nodeName:}" failed. No retries permitted until 2026-09-16 13:03:52.181602268 +0000 UTC m=+939.374250834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d516bad0-cbd0-4c98-8dd1-729170b4d151-proxy-tls") pod "success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" (UID: "d516bad0-cbd0-4c98-8dd1-729170b4d151") : secret "success-200-isvc-03a3a-predictor-serving-cert" not found Sep 16 13:03:51.682003 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.681985 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:51.690087 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:51.690062 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsx6c\" (UniqueName: \"kubernetes.io/projected/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-api-access-rsx6c\") pod \"success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:52.185246 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:52.185200 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d516bad0-cbd0-4c98-8dd1-729170b4d151-proxy-tls\") pod \"success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:52.187564 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:52.187533 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d516bad0-cbd0-4c98-8dd1-729170b4d151-proxy-tls\") pod \"success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:52.353406 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:52.353372 2584 generic.go:356] "Generic (PLEG): container finished" podID="d0633927-e158-43da-a712-ee87d0caa672" containerID="b73756b9422f0b47cdbbf0266393bc08a8b0603bd98540c300dc17253b535828" exitCode=2 Sep 16 13:03:52.353749 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:52.353450 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" event={"ID":"d0633927-e158-43da-a712-ee87d0caa672","Type":"ContainerDied","Data":"b73756b9422f0b47cdbbf0266393bc08a8b0603bd98540c300dc17253b535828"} Sep 16 13:03:52.401879 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:52.401849 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:52.672970 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:52.672937 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj"] Sep 16 13:03:52.676212 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:03:52.676183 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd516bad0_cbd0_4c98_8dd1_729170b4d151.slice/crio-7dde6f8dd4b2298dabf4fe6e0127b4fa06c2d156ec7d4d75d57408ea0a206603 WatchSource:0}: Error finding container 7dde6f8dd4b2298dabf4fe6e0127b4fa06c2d156ec7d4d75d57408ea0a206603: Status 404 returned error can't find the container with id 7dde6f8dd4b2298dabf4fe6e0127b4fa06c2d156ec7d4d75d57408ea0a206603 Sep 16 13:03:53.025015 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:53.024976 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused" Sep 16 13:03:53.029420 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:53.029391 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Sep 16 13:03:53.272293 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:53.272259 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:03:53.360227 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:53.360125 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" event={"ID":"d516bad0-cbd0-4c98-8dd1-729170b4d151","Type":"ContainerStarted","Data":"77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57"} Sep 16 13:03:53.360227 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:53.360206 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" event={"ID":"d516bad0-cbd0-4c98-8dd1-729170b4d151","Type":"ContainerStarted","Data":"66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32"} Sep 16 13:03:53.360227 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:53.360230 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" event={"ID":"d516bad0-cbd0-4c98-8dd1-729170b4d151","Type":"ContainerStarted","Data":"7dde6f8dd4b2298dabf4fe6e0127b4fa06c2d156ec7d4d75d57408ea0a206603"} Sep 16 13:03:53.360882 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:53.360253 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:53.360882 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:53.360281 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:53.361555 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:53.361526 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Sep 16 13:03:53.380999 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:53.380957 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podStartSLOduration=2.380943344 podStartE2EDuration="2.380943344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:03:53.379844633 +0000 UTC m=+940.572493232" watchObservedRunningTime="2026-09-16 13:03:53.380943344 +0000 UTC m=+940.573591933" Sep 16 13:03:54.366108 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.366079 2584 generic.go:356] "Generic (PLEG): container finished" podID="d0633927-e158-43da-a712-ee87d0caa672" containerID="ba31021f2cc699b32bcdfdd97b35f51a1f64fc040eb10b8cef4ec5e81c60343a" exitCode=0 Sep 16 13:03:54.366477 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.366153 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" event={"ID":"d0633927-e158-43da-a712-ee87d0caa672","Type":"ContainerDied","Data":"ba31021f2cc699b32bcdfdd97b35f51a1f64fc040eb10b8cef4ec5e81c60343a"} Sep 16 13:03:54.366477 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.366439 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Sep 16 13:03:54.421952 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.421928 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:03:54.505813 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.505784 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0633927-e158-43da-a712-ee87d0caa672-proxy-tls\") pod \"d0633927-e158-43da-a712-ee87d0caa672\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " Sep 16 13:03:54.505960 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.505866 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0633927-e158-43da-a712-ee87d0caa672-kube-rbac-proxy-sar-config\") pod \"d0633927-e158-43da-a712-ee87d0caa672\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " Sep 16 13:03:54.506010 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.505967 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f59nx\" (UniqueName: \"kubernetes.io/projected/d0633927-e158-43da-a712-ee87d0caa672-kube-api-access-f59nx\") pod \"d0633927-e158-43da-a712-ee87d0caa672\" (UID: \"d0633927-e158-43da-a712-ee87d0caa672\") " Sep 16 13:03:54.506143 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.506123 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0633927-e158-43da-a712-ee87d0caa672-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0633927-e158-43da-a712-ee87d0caa672" (UID: "d0633927-e158-43da-a712-ee87d0caa672"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:03:54.506252 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.506236 2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0633927-e158-43da-a712-ee87d0caa672-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:03:54.507981 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.507961 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0633927-e158-43da-a712-ee87d0caa672-kube-api-access-f59nx" (OuterVolumeSpecName: "kube-api-access-f59nx") pod "d0633927-e158-43da-a712-ee87d0caa672" (UID: "d0633927-e158-43da-a712-ee87d0caa672"). InnerVolumeSpecName "kube-api-access-f59nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:03:54.508058 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.508043 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0633927-e158-43da-a712-ee87d0caa672-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0633927-e158-43da-a712-ee87d0caa672" (UID: "d0633927-e158-43da-a712-ee87d0caa672"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:03:54.607418 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.607387 2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0633927-e158-43da-a712-ee87d0caa672-proxy-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:03:54.607418 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:54.607413 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59nx\" (UniqueName: \"kubernetes.io/projected/d0633927-e158-43da-a712-ee87d0caa672-kube-api-access-f59nx\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:03:55.372303 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:55.372273 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" Sep 16 13:03:55.372670 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:55.372311 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8" event={"ID":"d0633927-e158-43da-a712-ee87d0caa672","Type":"ContainerDied","Data":"ff2e68470591a01deac7e42166ba2f61e8c0ec49465a8b383d9596ddff4e6a73"} Sep 16 13:03:55.372670 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:55.372380 2584 scope.go:117] "RemoveContainer" containerID="b73756b9422f0b47cdbbf0266393bc08a8b0603bd98540c300dc17253b535828" Sep 16 13:03:55.381561 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:55.381546 2584 scope.go:117] "RemoveContainer" containerID="ba31021f2cc699b32bcdfdd97b35f51a1f64fc040eb10b8cef4ec5e81c60343a" Sep 16 13:03:55.395257 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:55.395232 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8"] Sep 16 13:03:55.397119 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:55.397096 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18839-predictor-7978f4cd9c-nchb8"] Sep 16 13:03:55.473027 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:55.472981 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0633927-e158-43da-a712-ee87d0caa672" path="/var/lib/kubelet/pods/d0633927-e158-43da-a712-ee87d0caa672/volumes" Sep 16 13:03:59.372003 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:59.371970 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:03:59.372616 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:03:59.372587 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Sep 16 13:04:09.372788 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:09.372750 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Sep 16 13:04:19.372900 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:19.372860 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Sep 16 13:04:29.373580 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:29.373544 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Sep 16 13:04:39.374382 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:39.374299 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:04:41.146741 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:41.146708 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:04:41.417961 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:41.417876 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:04:51.281640 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.281601 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:04:51.379276 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.379245 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:04:51.485505 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.485476 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:04:51.491751 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.491702 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx"] Sep 16 13:04:51.492051 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.492039 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kube-rbac-proxy" Sep 16 13:04:51.492105 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.492052 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kube-rbac-proxy" Sep 16 13:04:51.492105 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.492065 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" Sep 16 13:04:51.492105 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.492070 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" Sep 16 13:04:51.492247 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.492121 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kube-rbac-proxy" Sep 16 13:04:51.492247 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.492130 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0633927-e158-43da-a712-ee87d0caa672" containerName="kserve-container" Sep 16 13:04:51.494509 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.494486 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.497082 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.497056 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-46b60-predictor-serving-cert\"" Sep 16 13:04:51.497219 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.497120 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-46b60-kube-rbac-proxy-sar-config\"" Sep 16 13:04:51.507396 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.507373 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx"] Sep 16 13:04:51.582792 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.582717 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:04:51.669150 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.669114 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-46b60-predictor-5894d887c5-kbrpx\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.669355 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.669185 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2b6\" (UniqueName: \"kubernetes.io/projected/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-api-access-qw2b6\") pod \"success-200-isvc-46b60-predictor-5894d887c5-kbrpx\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.669355 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.669277 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-proxy-tls\") pod \"success-200-isvc-46b60-predictor-5894d887c5-kbrpx\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.680212 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.680176 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:04:51.769975 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.769940 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-proxy-tls\") pod \"success-200-isvc-46b60-predictor-5894d887c5-kbrpx\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.770139 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.770006 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-46b60-predictor-5894d887c5-kbrpx\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.770139 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.770040 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2b6\" (UniqueName: \"kubernetes.io/projected/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-api-access-qw2b6\") pod \"success-200-isvc-46b60-predictor-5894d887c5-kbrpx\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.770763 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.770742 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-46b60-predictor-5894d887c5-kbrpx\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.772439 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.772421 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-proxy-tls\") pod \"success-200-isvc-46b60-predictor-5894d887c5-kbrpx\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.777852 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.777832 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2b6\" (UniqueName: \"kubernetes.io/projected/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-api-access-qw2b6\") pod \"success-200-isvc-46b60-predictor-5894d887c5-kbrpx\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.779798 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.779777 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:04:51.807102 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.807077 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:51.836582 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:51.836511 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:04:52.077819 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:52.077794 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx"] Sep 16 13:04:52.079583 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:04:52.079555 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb30dd14_22b0_4fe1_a5b7_d4d2f578bad8.slice/crio-d83e4a52e91554bc1d073801d198f1f4507a67a55c96b6766a399733407233aa WatchSource:0}: Error finding container d83e4a52e91554bc1d073801d198f1f4507a67a55c96b6766a399733407233aa: Status 404 returned error can't find the container with id d83e4a52e91554bc1d073801d198f1f4507a67a55c96b6766a399733407233aa Sep 16 13:04:52.634142 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:52.634108 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" event={"ID":"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8","Type":"ContainerStarted","Data":"aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0"} Sep 16 13:04:52.634634 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:52.634190 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" event={"ID":"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8","Type":"ContainerStarted","Data":"d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d"} Sep 16 13:04:52.634634 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:52.634222 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" event={"ID":"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8","Type":"ContainerStarted","Data":"d83e4a52e91554bc1d073801d198f1f4507a67a55c96b6766a399733407233aa"} Sep 16 13:04:52.634634 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:52.634259 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:52.653155 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:52.653106 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" podStartSLOduration=1.653089459 podStartE2EDuration="1.653089459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:04:52.651232768 +0000 UTC m=+999.843881378" watchObservedRunningTime="2026-09-16 13:04:52.653089459 +0000 UTC m=+999.845738044" Sep 16 13:04:53.045678 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:53.045644 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:04:53.638840 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:53.638806 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:53.640305 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:53.640274 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Sep 16 13:04:54.643243 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:54.643202 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Sep 16 13:04:59.647901 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:59.647870 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:04:59.648387 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:04:59.648360 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Sep 16 13:05:09.648957 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:05:09.648918 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Sep 16 13:05:19.648403 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:05:19.648361 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Sep 16 13:05:29.648469 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:05:29.648429 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Sep 16 13:05:39.650093 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:05:39.650063 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:05:51.471276 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:05:51.471233 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:05:51.574324 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:05:51.574290 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:06:16.387093 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:06:16.387050 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:06:58.201652 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:06:58.201596 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:07:18.863534 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:07:18.863500 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:07:18.921389 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:07:18.921356 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:08:13.332507 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:08:13.332477 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:08:13.333584 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:08:13.333564 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:08:13.347425 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:08:13.347400 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:08:13.348517 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:08:13.348494 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:11:43.223107 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:11:43.223071 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:12:03.450417 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:12:03.450380 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:06.327285 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.327254 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:06.455154 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.455121 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj"] Sep 16 13:13:06.455451 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.455391 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" containerID="cri-o://66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32" gracePeriod=30 Sep 16 13:13:06.455548 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.455398 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kube-rbac-proxy" containerID="cri-o://77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57" gracePeriod=30 Sep 16 13:13:06.465305 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.465276 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:06.500995 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.500963 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw"] Sep 16 13:13:06.504608 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.504587 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.507447 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.507425 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bfe0b-predictor-serving-cert\"" Sep 16 13:13:06.507560 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.507478 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bfe0b-kube-rbac-proxy-sar-config\"" Sep 16 13:13:06.514495 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.514467 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw"] Sep 16 13:13:06.542044 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.542020 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:06.605326 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.605255 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:06.619450 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.619419 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bfe0b-predictor-85fd74875c-765tw\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.619583 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.619486 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a991cf08-d4aa-4fbd-93b7-74abd42a0725-proxy-tls\") pod \"success-200-isvc-bfe0b-predictor-85fd74875c-765tw\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.619583 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.619515 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b749z\" (UniqueName: \"kubernetes.io/projected/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-api-access-b749z\") pod \"success-200-isvc-bfe0b-predictor-85fd74875c-765tw\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.684850 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.684820 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:06.720427 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.720399 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bfe0b-predictor-85fd74875c-765tw\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.720562 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.720444 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a991cf08-d4aa-4fbd-93b7-74abd42a0725-proxy-tls\") pod \"success-200-isvc-bfe0b-predictor-85fd74875c-765tw\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.720562 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.720464 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b749z\" (UniqueName: \"kubernetes.io/projected/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-api-access-b749z\") pod \"success-200-isvc-bfe0b-predictor-85fd74875c-765tw\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.721150 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.721127 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bfe0b-predictor-85fd74875c-765tw\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.722877 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.722856 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a991cf08-d4aa-4fbd-93b7-74abd42a0725-proxy-tls\") pod \"success-200-isvc-bfe0b-predictor-85fd74875c-765tw\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.729760 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.729740 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b749z\" (UniqueName: \"kubernetes.io/projected/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-api-access-b749z\") pod \"success-200-isvc-bfe0b-predictor-85fd74875c-765tw\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.784797 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.784767 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:06.815749 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.815713 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:06.831064 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.831032 2584 generic.go:356] "Generic (PLEG): container finished" podID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerID="77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57" exitCode=2 Sep 16 13:13:06.831149 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:06.831065 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" event={"ID":"d516bad0-cbd0-4c98-8dd1-729170b4d151","Type":"ContainerDied","Data":"77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57"} Sep 16 13:13:07.090771 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:07.090746 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw"] Sep 16 13:13:07.093004 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:13:07.092980 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda991cf08_d4aa_4fbd_93b7_74abd42a0725.slice/crio-8b54a6fe4ffa5bece0a308557417c2b5fe31a8731f49d1727f1dd8d519e68795 WatchSource:0}: Error finding container 8b54a6fe4ffa5bece0a308557417c2b5fe31a8731f49d1727f1dd8d519e68795: Status 404 returned error can't find the container with id 8b54a6fe4ffa5bece0a308557417c2b5fe31a8731f49d1727f1dd8d519e68795 Sep 16 13:13:07.837299 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:07.837262 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" event={"ID":"a991cf08-d4aa-4fbd-93b7-74abd42a0725","Type":"ContainerStarted","Data":"9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46"} Sep 16 13:13:07.837726 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:07.837318 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" event={"ID":"a991cf08-d4aa-4fbd-93b7-74abd42a0725","Type":"ContainerStarted","Data":"a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02"} Sep 16 13:13:07.837726 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:07.837345 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" event={"ID":"a991cf08-d4aa-4fbd-93b7-74abd42a0725","Type":"ContainerStarted","Data":"8b54a6fe4ffa5bece0a308557417c2b5fe31a8731f49d1727f1dd8d519e68795"} Sep 16 13:13:07.837726 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:07.837370 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:07.837726 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:07.837455 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:07.838933 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:07.838898 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Sep 16 13:13:07.856365 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:07.856316 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" podStartSLOduration=1.856303077 podStartE2EDuration="1.856303077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:07.854738522 +0000 UTC m=+1495.047387134" watchObservedRunningTime="2026-09-16 13:13:07.856303077 +0000 UTC m=+1495.048951666" Sep 16 13:13:08.841819 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:08.841781 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Sep 16 13:13:09.367656 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.367612 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Sep 16 13:13:09.372983 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.372958 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Sep 16 13:13:09.502984 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.502960 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:13:09.642855 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.642771 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d516bad0-cbd0-4c98-8dd1-729170b4d151-proxy-tls\") pod \"d516bad0-cbd0-4c98-8dd1-729170b4d151\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " Sep 16 13:13:09.642855 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.642833 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-rbac-proxy-sar-config\") pod \"d516bad0-cbd0-4c98-8dd1-729170b4d151\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " Sep 16 13:13:09.643080 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.642885 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsx6c\" (UniqueName: \"kubernetes.io/projected/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-api-access-rsx6c\") pod \"d516bad0-cbd0-4c98-8dd1-729170b4d151\" (UID: \"d516bad0-cbd0-4c98-8dd1-729170b4d151\") " Sep 16 13:13:09.643262 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.643232 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d516bad0-cbd0-4c98-8dd1-729170b4d151" (UID: "d516bad0-cbd0-4c98-8dd1-729170b4d151"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:13:09.644801 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.644774 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-api-access-rsx6c" (OuterVolumeSpecName: "kube-api-access-rsx6c") pod "d516bad0-cbd0-4c98-8dd1-729170b4d151" (UID: "d516bad0-cbd0-4c98-8dd1-729170b4d151"). InnerVolumeSpecName "kube-api-access-rsx6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:13:09.644904 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.644839 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d516bad0-cbd0-4c98-8dd1-729170b4d151-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d516bad0-cbd0-4c98-8dd1-729170b4d151" (UID: "d516bad0-cbd0-4c98-8dd1-729170b4d151"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:13:09.743523 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.743487 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsx6c\" (UniqueName: \"kubernetes.io/projected/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-api-access-rsx6c\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:13:09.743523 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.743518 2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d516bad0-cbd0-4c98-8dd1-729170b4d151-proxy-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:13:09.743523 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.743528 2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d516bad0-cbd0-4c98-8dd1-729170b4d151-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:13:09.848461 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.848426 2584 generic.go:356] "Generic (PLEG): container finished" podID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerID="66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32" exitCode=0 Sep 16 13:13:09.848859 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.848482 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" event={"ID":"d516bad0-cbd0-4c98-8dd1-729170b4d151","Type":"ContainerDied","Data":"66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32"} Sep 16 13:13:09.848859 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.848485 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" Sep 16 13:13:09.848859 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.848532 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj" event={"ID":"d516bad0-cbd0-4c98-8dd1-729170b4d151","Type":"ContainerDied","Data":"7dde6f8dd4b2298dabf4fe6e0127b4fa06c2d156ec7d4d75d57408ea0a206603"} Sep 16 13:13:09.848859 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.848559 2584 scope.go:117] "RemoveContainer" containerID="77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57" Sep 16 13:13:09.857575 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.857555 2584 scope.go:117] "RemoveContainer" containerID="66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32" Sep 16 13:13:09.864843 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.864826 2584 scope.go:117] "RemoveContainer" containerID="77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57" Sep 16 13:13:09.865087 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:13:09.865069 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57\": container with ID starting with 77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57 not found: ID does not exist" containerID="77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57" Sep 16 13:13:09.865135 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.865097 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57"} err="failed to get container status \"77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57\": rpc error: code = NotFound desc = could not find container \"77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57\": container with ID starting with 77ea53261d824179780085e80445c8fc3137d51103c740400e856ffcbb683a57 not found: ID does not exist" Sep 16 13:13:09.865135 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.865113 2584 scope.go:117] "RemoveContainer" containerID="66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32" Sep 16 13:13:09.865403 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:13:09.865384 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32\": container with ID starting with 66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32 not found: ID does not exist" containerID="66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32" Sep 16 13:13:09.865461 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.865404 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32"} err="failed to get container status \"66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32\": rpc error: code = NotFound desc = could not find container \"66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32\": container with ID starting with 66ef0f3d3bd30ee6efefc8a7bd81928fbd7ce25a0b818d6828ea60e0a0650c32 not found: ID does not exist" Sep 16 13:13:09.870798 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.870778 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj"] Sep 16 13:13:09.875940 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:09.875922 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03a3a-predictor-7cf78bdcf-gk8dj"] Sep 16 13:13:11.472794 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:11.472762 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" path="/var/lib/kubelet/pods/d516bad0-cbd0-4c98-8dd1-729170b4d151/volumes" Sep 16 13:13:13.373951 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:13.373921 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:13:13.374979 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:13.374946 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:13:13.382209 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:13.382191 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:13:13.383261 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:13.382920 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:13:13.846739 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:13.846711 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:13:13.847193 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:13.847147 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Sep 16 13:13:23.847913 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:23.847874 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Sep 16 13:13:33.847388 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:33.847348 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Sep 16 13:13:43.848221 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:43.848108 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Sep 16 13:13:53.848240 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:13:53.848210 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:14:06.305348 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.305304 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx"] Sep 16 13:14:06.305847 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.305527 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kserve-container" containerID="cri-o://d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d" gracePeriod=30 Sep 16 13:14:06.305847 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.305602 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kube-rbac-proxy" containerID="cri-o://aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0" gracePeriod=30 Sep 16 13:14:06.408520 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.408481 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q"] Sep 16 13:14:06.408845 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.408832 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kube-rbac-proxy" Sep 16 13:14:06.408891 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.408846 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kube-rbac-proxy" Sep 16 13:14:06.408891 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.408867 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" Sep 16 13:14:06.408891 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.408872 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" Sep 16 13:14:06.408984 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.408923 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kube-rbac-proxy" Sep 16 13:14:06.408984 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.408930 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d516bad0-cbd0-4c98-8dd1-729170b4d151" containerName="kserve-container" Sep 16 13:14:06.412275 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.412252 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:06.416436 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.416417 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-aa957-kube-rbac-proxy-sar-config\"" Sep 16 13:14:06.416436 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.416424 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-aa957-predictor-serving-cert\"" Sep 16 13:14:06.423128 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.423104 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q"] Sep 16 13:14:06.505719 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.505042 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4t4\" (UniqueName: \"kubernetes.io/projected/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-api-access-fh4t4\") pod \"success-200-isvc-aa957-predictor-55cf84dbdf-bk89q\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:06.505719 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.505268 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-aa957-predictor-55cf84dbdf-bk89q\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:06.505719 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.505317 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/992c38ce-904e-4b40-a68f-7a8066332ec9-proxy-tls\") pod \"success-200-isvc-aa957-predictor-55cf84dbdf-bk89q\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:06.606555 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.606464 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-aa957-predictor-55cf84dbdf-bk89q\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:06.606555 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.606511 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/992c38ce-904e-4b40-a68f-7a8066332ec9-proxy-tls\") pod \"success-200-isvc-aa957-predictor-55cf84dbdf-bk89q\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:06.606783 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.606574 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4t4\" (UniqueName: \"kubernetes.io/projected/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-api-access-fh4t4\") pod \"success-200-isvc-aa957-predictor-55cf84dbdf-bk89q\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:06.607262 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.607238 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-aa957-predictor-55cf84dbdf-bk89q\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:06.609136 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.609104 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/992c38ce-904e-4b40-a68f-7a8066332ec9-proxy-tls\") pod \"success-200-isvc-aa957-predictor-55cf84dbdf-bk89q\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:06.617674 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.617639 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4t4\" (UniqueName: \"kubernetes.io/projected/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-api-access-fh4t4\") pod \"success-200-isvc-aa957-predictor-55cf84dbdf-bk89q\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:06.725149 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:06.725106 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:07.001244 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:07.001219 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q"] Sep 16 13:14:07.003263 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:14:07.003239 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992c38ce_904e_4b40_a68f_7a8066332ec9.slice/crio-8a1160dd4db679b89d2671992e35e7438e61ffcf688938d6af68a9aae2584cff WatchSource:0}: Error finding container 8a1160dd4db679b89d2671992e35e7438e61ffcf688938d6af68a9aae2584cff: Status 404 returned error can't find the container with id 8a1160dd4db679b89d2671992e35e7438e61ffcf688938d6af68a9aae2584cff Sep 16 13:14:07.106481 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:07.106437 2584 generic.go:356] "Generic (PLEG): container finished" podID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerID="aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0" exitCode=2 Sep 16 13:14:07.106610 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:07.106558 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" event={"ID":"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8","Type":"ContainerDied","Data":"aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0"} Sep 16 13:14:07.108409 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:07.108387 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" event={"ID":"992c38ce-904e-4b40-a68f-7a8066332ec9","Type":"ContainerStarted","Data":"cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326"} Sep 16 13:14:07.108522 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:07.108438 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" event={"ID":"992c38ce-904e-4b40-a68f-7a8066332ec9","Type":"ContainerStarted","Data":"8a1160dd4db679b89d2671992e35e7438e61ffcf688938d6af68a9aae2584cff"} Sep 16 13:14:07.229862 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:07.229828 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:07.285025 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:07.284948 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:08.114769 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:08.114734 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" event={"ID":"992c38ce-904e-4b40-a68f-7a8066332ec9","Type":"ContainerStarted","Data":"687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913"} Sep 16 13:14:08.115207 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:08.114827 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:08.115207 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:08.114944 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:08.116154 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:08.116128 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Sep 16 13:14:08.132792 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:08.132752 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" podStartSLOduration=2.132727396 podStartE2EDuration="2.132727396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:08.131561758 +0000 UTC m=+1555.324210360" watchObservedRunningTime="2026-09-16 13:14:08.132727396 +0000 UTC m=+1555.325375984" Sep 16 13:14:09.119131 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.119089 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Sep 16 13:14:09.452816 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.452794 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:14:09.533514 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.533475 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw2b6\" (UniqueName: \"kubernetes.io/projected/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-api-access-qw2b6\") pod \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " Sep 16 13:14:09.533708 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.533526 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-rbac-proxy-sar-config\") pod \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " Sep 16 13:14:09.533708 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.533602 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-proxy-tls\") pod \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\" (UID: \"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8\") " Sep 16 13:14:09.533920 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.533890 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" (UID: "cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:14:09.535714 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.535690 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" (UID: "cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:14:09.535806 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.535710 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-api-access-qw2b6" (OuterVolumeSpecName: "kube-api-access-qw2b6") pod "cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" (UID: "cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8"). InnerVolumeSpecName "kube-api-access-qw2b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:14:09.634966 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.634890 2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-proxy-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:14:09.634966 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.634919 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw2b6\" (UniqueName: \"kubernetes.io/projected/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-api-access-qw2b6\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:14:09.634966 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:09.634929 2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:14:10.124302 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.124270 2584 generic.go:356] "Generic (PLEG): container finished" podID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerID="d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d" exitCode=0 Sep 16 13:14:10.124693 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.124321 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" Sep 16 13:14:10.124693 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.124355 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" event={"ID":"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8","Type":"ContainerDied","Data":"d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d"} Sep 16 13:14:10.124693 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.124414 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx" event={"ID":"cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8","Type":"ContainerDied","Data":"d83e4a52e91554bc1d073801d198f1f4507a67a55c96b6766a399733407233aa"} Sep 16 13:14:10.124693 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.124440 2584 scope.go:117] "RemoveContainer" containerID="aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0" Sep 16 13:14:10.133400 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.133384 2584 scope.go:117] "RemoveContainer" containerID="d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d" Sep 16 13:14:10.140690 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.140673 2584 scope.go:117] "RemoveContainer" containerID="aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0" Sep 16 13:14:10.140975 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:14:10.140953 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0\": container with ID starting with aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0 not found: ID does not exist" containerID="aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0" Sep 16 13:14:10.141054 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.140981 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0"} err="failed to get container status \"aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0\": rpc error: code = NotFound desc = could not find container \"aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0\": container with ID starting with aa53e31d771e1a4dda55d62fb0c6b82059109995d9933d8fd886b82b5ff4f1e0 not found: ID does not exist" Sep 16 13:14:10.141054 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.140998 2584 scope.go:117] "RemoveContainer" containerID="d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d" Sep 16 13:14:10.141267 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:14:10.141248 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d\": container with ID starting with d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d not found: ID does not exist" containerID="d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d" Sep 16 13:14:10.141361 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.141274 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d"} err="failed to get container status \"d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d\": rpc error: code = NotFound desc = could not find container \"d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d\": container with ID starting with d9d8732f95725a35d32443777ad34bc218c2091d5fc8ac793ffed3a0f02e2a7d not found: ID does not exist" Sep 16 13:14:10.145919 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.145897 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx"] Sep 16 13:14:10.149520 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:10.149490 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46b60-predictor-5894d887c5-kbrpx"] Sep 16 13:14:11.473008 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:11.472971 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" path="/var/lib/kubelet/pods/cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8/volumes" Sep 16 13:14:14.123679 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:14.123647 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:14.124214 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:14.124155 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Sep 16 13:14:16.608130 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.608057 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:16.668894 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.668857 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw"] Sep 16 13:14:16.669123 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.669100 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kserve-container" containerID="cri-o://a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02" gracePeriod=30 Sep 16 13:14:16.669250 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.669145 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kube-rbac-proxy" containerID="cri-o://9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46" gracePeriod=30 Sep 16 13:14:16.690130 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.690077 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:16.772869 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.772834 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:16.777997 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.777970 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb"] Sep 16 13:14:16.778355 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.778341 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kube-rbac-proxy" Sep 16 13:14:16.778438 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.778357 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kube-rbac-proxy" Sep 16 13:14:16.778438 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.778370 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kserve-container" Sep 16 13:14:16.778438 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.778376 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kserve-container" Sep 16 13:14:16.778438 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.778436 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kserve-container" Sep 16 13:14:16.778624 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.778448 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb30dd14-22b0-4fe1-a5b7-d4d2f578bad8" containerName="kube-rbac-proxy" Sep 16 13:14:16.781517 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.781498 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:16.783923 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.783897 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d30d6-predictor-serving-cert\"" Sep 16 13:14:16.784273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.784253 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d30d6-kube-rbac-proxy-sar-config\"" Sep 16 13:14:16.788037 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.788016 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb"] Sep 16 13:14:16.836238 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.836210 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:16.890483 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.890395 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg7xm\" (UniqueName: \"kubernetes.io/projected/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-api-access-vg7xm\") pod \"success-200-isvc-d30d6-predictor-597df4949-lzkwb\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:16.890483 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.890436 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d30d6-predictor-597df4949-lzkwb\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:16.890664 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.890609 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-proxy-tls\") pod \"success-200-isvc-d30d6-predictor-597df4949-lzkwb\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:16.900440 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.900390 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:16.975459 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.975424 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:16.991263 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.991238 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-proxy-tls\") pod \"success-200-isvc-d30d6-predictor-597df4949-lzkwb\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:16.991429 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.991281 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg7xm\" (UniqueName: \"kubernetes.io/projected/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-api-access-vg7xm\") pod \"success-200-isvc-d30d6-predictor-597df4949-lzkwb\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:16.991429 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.991300 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d30d6-predictor-597df4949-lzkwb\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:16.991429 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:14:16.991416 2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-serving-cert: secret "success-200-isvc-d30d6-predictor-serving-cert" not found Sep 16 13:14:16.991652 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:14:16.991483 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-proxy-tls podName:12d32336-5ba8-4ab0-b7d1-d9b03e60629e nodeName:}" failed. No retries permitted until 2026-09-16 13:14:17.491464589 +0000 UTC m=+1564.684113179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-proxy-tls") pod "success-200-isvc-d30d6-predictor-597df4949-lzkwb" (UID: "12d32336-5ba8-4ab0-b7d1-d9b03e60629e") : secret "success-200-isvc-d30d6-predictor-serving-cert" not found Sep 16 13:14:16.991918 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:16.991900 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d30d6-predictor-597df4949-lzkwb\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:17.002003 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:17.001982 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg7xm\" (UniqueName: \"kubernetes.io/projected/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-api-access-vg7xm\") pod \"success-200-isvc-d30d6-predictor-597df4949-lzkwb\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:17.110222 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:17.110155 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:17.158207 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:17.158125 2584 generic.go:356] "Generic (PLEG): container finished" podID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerID="9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46" exitCode=2 Sep 16 13:14:17.158207 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:17.158189 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" event={"ID":"a991cf08-d4aa-4fbd-93b7-74abd42a0725","Type":"ContainerDied","Data":"9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46"} Sep 16 13:14:17.495407 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:17.495377 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-proxy-tls\") pod \"success-200-isvc-d30d6-predictor-597df4949-lzkwb\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:17.497884 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:17.497854 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-proxy-tls\") pod \"success-200-isvc-d30d6-predictor-597df4949-lzkwb\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:17.693878 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:17.693845 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:17.965649 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:17.965624 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb"] Sep 16 13:14:17.967487 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:14:17.967453 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d32336_5ba8_4ab0_b7d1_d9b03e60629e.slice/crio-5c58a7075f0ee7146558440bf1307d77a8915f0f70affc1b8da0aed9b8936127 WatchSource:0}: Error finding container 5c58a7075f0ee7146558440bf1307d77a8915f0f70affc1b8da0aed9b8936127: Status 404 returned error can't find the container with id 5c58a7075f0ee7146558440bf1307d77a8915f0f70affc1b8da0aed9b8936127 Sep 16 13:14:18.163957 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:18.163927 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" event={"ID":"12d32336-5ba8-4ab0-b7d1-d9b03e60629e","Type":"ContainerStarted","Data":"912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26"} Sep 16 13:14:18.164108 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:18.163990 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" event={"ID":"12d32336-5ba8-4ab0-b7d1-d9b03e60629e","Type":"ContainerStarted","Data":"609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262"} Sep 16 13:14:18.164108 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:18.164014 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" event={"ID":"12d32336-5ba8-4ab0-b7d1-d9b03e60629e","Type":"ContainerStarted","Data":"5c58a7075f0ee7146558440bf1307d77a8915f0f70affc1b8da0aed9b8936127"} Sep 16 13:14:18.164108 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:18.164037 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:18.164108 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:18.164067 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:18.165327 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:18.165302 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Sep 16 13:14:18.182301 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:18.182256 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podStartSLOduration=2.182240951 podStartE2EDuration="2.182240951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:18.180703246 +0000 UTC m=+1565.373351834" watchObservedRunningTime="2026-09-16 13:14:18.182240951 +0000 UTC m=+1565.374889539" Sep 16 13:14:18.842096 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:18.842054 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused" Sep 16 13:14:19.168324 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.168232 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Sep 16 13:14:19.723516 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.723492 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:14:19.810962 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.810883 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-rbac-proxy-sar-config\") pod \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " Sep 16 13:14:19.810962 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.810926 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a991cf08-d4aa-4fbd-93b7-74abd42a0725-proxy-tls\") pod \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " Sep 16 13:14:19.811133 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.810995 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b749z\" (UniqueName: \"kubernetes.io/projected/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-api-access-b749z\") pod \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\" (UID: \"a991cf08-d4aa-4fbd-93b7-74abd42a0725\") " Sep 16 13:14:19.811281 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.811256 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a991cf08-d4aa-4fbd-93b7-74abd42a0725" (UID: "a991cf08-d4aa-4fbd-93b7-74abd42a0725"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:14:19.812964 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.812937 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-api-access-b749z" (OuterVolumeSpecName: "kube-api-access-b749z") pod "a991cf08-d4aa-4fbd-93b7-74abd42a0725" (UID: "a991cf08-d4aa-4fbd-93b7-74abd42a0725"). InnerVolumeSpecName "kube-api-access-b749z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:14:19.813068 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.813022 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a991cf08-d4aa-4fbd-93b7-74abd42a0725-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a991cf08-d4aa-4fbd-93b7-74abd42a0725" (UID: "a991cf08-d4aa-4fbd-93b7-74abd42a0725"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:14:19.912234 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.912204 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b749z\" (UniqueName: \"kubernetes.io/projected/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-api-access-b749z\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:14:19.912234 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.912229 2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a991cf08-d4aa-4fbd-93b7-74abd42a0725-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:14:19.912234 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:19.912239 2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a991cf08-d4aa-4fbd-93b7-74abd42a0725-proxy-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:14:20.174720 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.174639 2584 generic.go:356] "Generic (PLEG): container finished" podID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerID="a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02" exitCode=0 Sep 16 13:14:20.174720 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.174686 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" Sep 16 13:14:20.174720 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.174717 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" event={"ID":"a991cf08-d4aa-4fbd-93b7-74abd42a0725","Type":"ContainerDied","Data":"a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02"} Sep 16 13:14:20.174945 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.174787 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw" event={"ID":"a991cf08-d4aa-4fbd-93b7-74abd42a0725","Type":"ContainerDied","Data":"8b54a6fe4ffa5bece0a308557417c2b5fe31a8731f49d1727f1dd8d519e68795"} Sep 16 13:14:20.174945 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.174815 2584 scope.go:117] "RemoveContainer" containerID="9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46" Sep 16 13:14:20.183219 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.183200 2584 scope.go:117] "RemoveContainer" containerID="a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02" Sep 16 13:14:20.190354 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.190332 2584 scope.go:117] "RemoveContainer" containerID="9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46" Sep 16 13:14:20.190619 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:14:20.190602 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46\": container with ID starting with 9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46 not found: ID does not exist" containerID="9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46" Sep 16 13:14:20.190671 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.190626 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46"} err="failed to get container status \"9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46\": rpc error: code = NotFound desc = could not find container \"9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46\": container with ID starting with 9b87b30faebf649ac4c79bb3cae084119b94f76efb83f3731236e98273032d46 not found: ID does not exist" Sep 16 13:14:20.190671 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.190642 2584 scope.go:117] "RemoveContainer" containerID="a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02" Sep 16 13:14:20.190887 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:14:20.190867 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02\": container with ID starting with a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02 not found: ID does not exist" containerID="a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02" Sep 16 13:14:20.190943 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.190896 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02"} err="failed to get container status \"a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02\": rpc error: code = NotFound desc = could not find container \"a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02\": container with ID starting with a0ab860a941b2a26439e0f74ef9f3c2ba4fdbc552aa2076731d6491143eddf02 not found: ID does not exist" Sep 16 13:14:20.196318 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.196293 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw"] Sep 16 13:14:20.197907 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:20.197885 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bfe0b-predictor-85fd74875c-765tw"] Sep 16 13:14:21.473129 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:21.473096 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" path="/var/lib/kubelet/pods/a991cf08-d4aa-4fbd-93b7-74abd42a0725/volumes" Sep 16 13:14:24.124680 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:24.124643 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Sep 16 13:14:24.172568 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:24.172547 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:14:24.173081 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:24.173057 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Sep 16 13:14:34.124563 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:34.124521 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Sep 16 13:14:34.173533 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:34.173497 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Sep 16 13:14:44.124381 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:44.124343 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Sep 16 13:14:44.173938 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:44.173904 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Sep 16 13:14:48.452377 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:48.452330 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:54.125103 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:54.125073 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:14:54.173439 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:14:54.173403 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Sep 16 13:15:04.174189 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:04.174139 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:15:06.438543 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:06.438467 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:06.643171 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:06.643133 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:16.537861 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.537765 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:16.598134 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.598104 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q"] Sep 16 13:15:16.598393 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.598362 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kserve-container" containerID="cri-o://cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326" gracePeriod=30 Sep 16 13:15:16.598612 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.598551 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kube-rbac-proxy" containerID="cri-o://687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913" gracePeriod=30 Sep 16 13:15:16.625536 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.625504 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:16.698015 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.697978 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:16.720692 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.720665 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd"] Sep 16 13:15:16.721077 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.721060 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kserve-container" Sep 16 13:15:16.721171 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.721081 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kserve-container" Sep 16 13:15:16.721171 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.721114 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kube-rbac-proxy" Sep 16 13:15:16.721171 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.721123 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kube-rbac-proxy" Sep 16 13:15:16.721347 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.721220 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kserve-container" Sep 16 13:15:16.721347 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.721234 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a991cf08-d4aa-4fbd-93b7-74abd42a0725" containerName="kube-rbac-proxy" Sep 16 13:15:16.724400 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.724379 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:16.726986 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.726964 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4c3c9-kube-rbac-proxy-sar-config\"" Sep 16 13:15:16.726986 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.726978 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4c3c9-predictor-serving-cert\"" Sep 16 13:15:16.733430 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.733410 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd"] Sep 16 13:15:16.775838 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.775807 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwcdf\" (UniqueName: \"kubernetes.io/projected/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-api-access-qwcdf\") pod \"success-200-isvc-4c3c9-predictor-58d5c86759-mspsd\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:16.776003 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.775852 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7940434-a9d6-453f-a98a-5052d19b3eb2-proxy-tls\") pod \"success-200-isvc-4c3c9-predictor-58d5c86759-mspsd\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:16.776066 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.776015 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4c3c9-predictor-58d5c86759-mspsd\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:16.801598 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.801519 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:16.876840 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.876800 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwcdf\" (UniqueName: \"kubernetes.io/projected/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-api-access-qwcdf\") pod \"success-200-isvc-4c3c9-predictor-58d5c86759-mspsd\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:16.877008 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.876858 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7940434-a9d6-453f-a98a-5052d19b3eb2-proxy-tls\") pod \"success-200-isvc-4c3c9-predictor-58d5c86759-mspsd\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:16.877008 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.876930 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4c3c9-predictor-58d5c86759-mspsd\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:16.877131 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:15:16.877046 2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-serving-cert: secret "success-200-isvc-4c3c9-predictor-serving-cert" not found Sep 16 13:15:16.877228 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:15:16.877126 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7940434-a9d6-453f-a98a-5052d19b3eb2-proxy-tls podName:a7940434-a9d6-453f-a98a-5052d19b3eb2 nodeName:}" failed. No retries permitted until 2026-09-16 13:15:17.377104897 +0000 UTC m=+1624.569753466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a7940434-a9d6-453f-a98a-5052d19b3eb2-proxy-tls") pod "success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" (UID: "a7940434-a9d6-453f-a98a-5052d19b3eb2") : secret "success-200-isvc-4c3c9-predictor-serving-cert" not found Sep 16 13:15:16.877792 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.877771 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4c3c9-predictor-58d5c86759-mspsd\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:16.885864 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.885838 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwcdf\" (UniqueName: \"kubernetes.io/projected/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-api-access-qwcdf\") pod \"success-200-isvc-4c3c9-predictor-58d5c86759-mspsd\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:16.899408 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:16.899377 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:17.000807 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:17.000777 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:17.229622 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:17.229591 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:17.318438 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:17.318413 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:17.382294 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:17.382268 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7940434-a9d6-453f-a98a-5052d19b3eb2-proxy-tls\") pod \"success-200-isvc-4c3c9-predictor-58d5c86759-mspsd\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:17.384583 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:17.384564 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7940434-a9d6-453f-a98a-5052d19b3eb2-proxy-tls\") pod \"success-200-isvc-4c3c9-predictor-58d5c86759-mspsd\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:17.456956 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:17.456926 2584 generic.go:356] "Generic (PLEG): container finished" podID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerID="687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913" exitCode=2 Sep 16 13:15:17.457087 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:17.456986 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" event={"ID":"992c38ce-904e-4b40-a68f-7a8066332ec9","Type":"ContainerDied","Data":"687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913"} Sep 16 13:15:17.635434 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:17.635348 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:17.905758 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:17.905732 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd"] Sep 16 13:15:17.907761 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:15:17.907727 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7940434_a9d6_453f_a98a_5052d19b3eb2.slice/crio-eb1a290929b489e7b94dbd66f71e008beae4ec323d4ed18b2c3fdb9576e90f72 WatchSource:0}: Error finding container eb1a290929b489e7b94dbd66f71e008beae4ec323d4ed18b2c3fdb9576e90f72: Status 404 returned error can't find the container with id eb1a290929b489e7b94dbd66f71e008beae4ec323d4ed18b2c3fdb9576e90f72 Sep 16 13:15:18.464057 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:18.464016 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" event={"ID":"a7940434-a9d6-453f-a98a-5052d19b3eb2","Type":"ContainerStarted","Data":"fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768"} Sep 16 13:15:18.464280 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:18.464081 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" event={"ID":"a7940434-a9d6-453f-a98a-5052d19b3eb2","Type":"ContainerStarted","Data":"52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b"} Sep 16 13:15:18.464280 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:18.464187 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:18.464280 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:18.464244 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:18.464280 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:18.464272 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" event={"ID":"a7940434-a9d6-453f-a98a-5052d19b3eb2","Type":"ContainerStarted","Data":"eb1a290929b489e7b94dbd66f71e008beae4ec323d4ed18b2c3fdb9576e90f72"} Sep 16 13:15:18.465666 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:18.465645 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Sep 16 13:15:18.481688 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:18.481644 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podStartSLOduration=2.481631202 podStartE2EDuration="2.481631202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:18.479872883 +0000 UTC m=+1625.672521517" watchObservedRunningTime="2026-09-16 13:15:18.481631202 +0000 UTC m=+1625.674279838" Sep 16 13:15:19.120230 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.120186 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused" Sep 16 13:15:19.468634 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.468589 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Sep 16 13:15:19.650954 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.650928 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:15:19.705218 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.705191 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/992c38ce-904e-4b40-a68f-7a8066332ec9-proxy-tls\") pod \"992c38ce-904e-4b40-a68f-7a8066332ec9\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " Sep 16 13:15:19.705350 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.705280 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-rbac-proxy-sar-config\") pod \"992c38ce-904e-4b40-a68f-7a8066332ec9\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " Sep 16 13:15:19.705350 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.705342 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh4t4\" (UniqueName: \"kubernetes.io/projected/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-api-access-fh4t4\") pod \"992c38ce-904e-4b40-a68f-7a8066332ec9\" (UID: \"992c38ce-904e-4b40-a68f-7a8066332ec9\") " Sep 16 13:15:19.705687 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.705660 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "992c38ce-904e-4b40-a68f-7a8066332ec9" (UID: "992c38ce-904e-4b40-a68f-7a8066332ec9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:15:19.707313 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.707285 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-api-access-fh4t4" (OuterVolumeSpecName: "kube-api-access-fh4t4") pod "992c38ce-904e-4b40-a68f-7a8066332ec9" (UID: "992c38ce-904e-4b40-a68f-7a8066332ec9"). InnerVolumeSpecName "kube-api-access-fh4t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:15:19.707313 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.707286 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992c38ce-904e-4b40-a68f-7a8066332ec9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "992c38ce-904e-4b40-a68f-7a8066332ec9" (UID: "992c38ce-904e-4b40-a68f-7a8066332ec9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:15:19.806812 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.806741 2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/992c38ce-904e-4b40-a68f-7a8066332ec9-proxy-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:15:19.806812 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.806765 2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:15:19.806812 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:19.806775 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh4t4\" (UniqueName: \"kubernetes.io/projected/992c38ce-904e-4b40-a68f-7a8066332ec9-kube-api-access-fh4t4\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:15:20.473977 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.473944 2584 generic.go:356] "Generic (PLEG): container finished" podID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerID="cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326" exitCode=0 Sep 16 13:15:20.474393 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.473996 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" Sep 16 13:15:20.474393 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.474022 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" event={"ID":"992c38ce-904e-4b40-a68f-7a8066332ec9","Type":"ContainerDied","Data":"cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326"} Sep 16 13:15:20.474393 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.474069 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q" event={"ID":"992c38ce-904e-4b40-a68f-7a8066332ec9","Type":"ContainerDied","Data":"8a1160dd4db679b89d2671992e35e7438e61ffcf688938d6af68a9aae2584cff"} Sep 16 13:15:20.474393 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.474096 2584 scope.go:117] "RemoveContainer" containerID="687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913" Sep 16 13:15:20.484188 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.484170 2584 scope.go:117] "RemoveContainer" containerID="cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326" Sep 16 13:15:20.491776 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.491759 2584 scope.go:117] "RemoveContainer" containerID="687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913" Sep 16 13:15:20.492018 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:15:20.492000 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913\": container with ID starting with 687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913 not found: ID does not exist" containerID="687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913" Sep 16 13:15:20.492081 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.492029 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913"} err="failed to get container status \"687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913\": rpc error: code = NotFound desc = could not find container \"687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913\": container with ID starting with 687ac9618a838e2cd273630b471a04518b55109a3708fd4668a465f1cfb07913 not found: ID does not exist" Sep 16 13:15:20.492081 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.492050 2584 scope.go:117] "RemoveContainer" containerID="cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326" Sep 16 13:15:20.492351 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:15:20.492331 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326\": container with ID starting with cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326 not found: ID does not exist" containerID="cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326" Sep 16 13:15:20.492406 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.492361 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326"} err="failed to get container status \"cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326\": rpc error: code = NotFound desc = could not find container \"cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326\": container with ID starting with cf54d07699f9f126dbf4b55cfb40fdb06fbf560837d6ace9d641a6542fd18326 not found: ID does not exist" Sep 16 13:15:20.496702 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.496679 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q"] Sep 16 13:15:20.500253 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:20.500234 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aa957-predictor-55cf84dbdf-bk89q"] Sep 16 13:15:21.473395 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:21.473358 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" path="/var/lib/kubelet/pods/992c38ce-904e-4b40-a68f-7a8066332ec9/volumes" Sep 16 13:15:24.472896 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:24.472866 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:15:24.473395 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:24.473246 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Sep 16 13:15:34.473919 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:34.473880 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Sep 16 13:15:44.474105 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:44.474063 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Sep 16 13:15:54.474252 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:15:54.474212 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Sep 16 13:16:04.475224 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:16:04.475192 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:16:16.767709 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:16:16.767675 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:17.327401 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:16:17.327372 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:33.004618 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:16:33.004578 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:13.404682 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:18:13.404651 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:18:13.405903 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:18:13.405883 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:18:13.412539 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:18:13.412519 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:18:13.413727 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:18:13.413707 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:20:58.510947 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:20:58.510908 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:07.357787 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:21:07.357700 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:07.428338 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:21:07.428298 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:13.434067 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:13.434036 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:23:13.436796 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:13.436773 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:23:13.442045 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:13.442026 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:23:13.445128 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:13.445109 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:23:31.542280 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.542249 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:31.604570 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.604531 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb"] Sep 16 13:23:31.604828 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.604788 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" containerID="cri-o://609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262" gracePeriod=30 Sep 16 13:23:31.604967 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.604834 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kube-rbac-proxy" containerID="cri-o://912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26" gracePeriod=30 Sep 16 13:23:31.629331 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.629310 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:31.735359 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.735332 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq"] Sep 16 13:23:31.735825 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.735795 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kube-rbac-proxy" Sep 16 13:23:31.735825 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.735818 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kube-rbac-proxy" Sep 16 13:23:31.736070 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.735857 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kserve-container" Sep 16 13:23:31.736070 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.735867 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kserve-container" Sep 16 13:23:31.736070 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.735981 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kube-rbac-proxy" Sep 16 13:23:31.736070 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.735996 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="992c38ce-904e-4b40-a68f-7a8066332ec9" containerName="kserve-container" Sep 16 13:23:31.736971 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.736948 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:31.739303 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.739287 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:31.741879 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.741862 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e1666-predictor-serving-cert\"" Sep 16 13:23:31.741977 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.741888 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e1666-kube-rbac-proxy-sar-config\"" Sep 16 13:23:31.751428 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.751402 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq"] Sep 16 13:23:31.811639 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.811520 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:31.834238 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.834209 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhfnv\" (UniqueName: \"kubernetes.io/projected/e041a187-a083-4a84-925b-aac987cd32db-kube-api-access-lhfnv\") pod \"success-200-isvc-e1666-predictor-6fcf567cf-z5ktq\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:31.834369 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.834251 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e041a187-a083-4a84-925b-aac987cd32db-proxy-tls\") pod \"success-200-isvc-e1666-predictor-6fcf567cf-z5ktq\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:31.834369 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.834320 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e041a187-a083-4a84-925b-aac987cd32db-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e1666-predictor-6fcf567cf-z5ktq\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:31.884948 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.884919 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:31.935239 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.935207 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e041a187-a083-4a84-925b-aac987cd32db-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e1666-predictor-6fcf567cf-z5ktq\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:31.935391 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.935274 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhfnv\" (UniqueName: \"kubernetes.io/projected/e041a187-a083-4a84-925b-aac987cd32db-kube-api-access-lhfnv\") pod \"success-200-isvc-e1666-predictor-6fcf567cf-z5ktq\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:31.935391 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.935314 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e041a187-a083-4a84-925b-aac987cd32db-proxy-tls\") pod \"success-200-isvc-e1666-predictor-6fcf567cf-z5ktq\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:31.935506 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:23:31.935438 2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e1666-predictor-serving-cert: secret "success-200-isvc-e1666-predictor-serving-cert" not found Sep 16 13:23:31.935558 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:23:31.935507 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e041a187-a083-4a84-925b-aac987cd32db-proxy-tls podName:e041a187-a083-4a84-925b-aac987cd32db nodeName:}" failed. No retries permitted until 2026-09-16 13:23:32.43548805 +0000 UTC m=+2119.628136622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e041a187-a083-4a84-925b-aac987cd32db-proxy-tls") pod "success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" (UID: "e041a187-a083-4a84-925b-aac987cd32db") : secret "success-200-isvc-e1666-predictor-serving-cert" not found Sep 16 13:23:31.935988 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.935967 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e041a187-a083-4a84-925b-aac987cd32db-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e1666-predictor-6fcf567cf-z5ktq\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:31.949350 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.949328 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhfnv\" (UniqueName: \"kubernetes.io/projected/e041a187-a083-4a84-925b-aac987cd32db-kube-api-access-lhfnv\") pod \"success-200-isvc-e1666-predictor-6fcf567cf-z5ktq\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:31.984489 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:31.984463 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:32.159023 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:32.158945 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:32.438808 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:32.438727 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e041a187-a083-4a84-925b-aac987cd32db-proxy-tls\") pod \"success-200-isvc-e1666-predictor-6fcf567cf-z5ktq\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:32.441120 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:32.441101 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e041a187-a083-4a84-925b-aac987cd32db-proxy-tls\") pod \"success-200-isvc-e1666-predictor-6fcf567cf-z5ktq\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:32.652465 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:32.652435 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:32.707686 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:32.707654 2584 generic.go:356] "Generic (PLEG): container finished" podID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerID="912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26" exitCode=2 Sep 16 13:23:32.707799 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:32.707740 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" event={"ID":"12d32336-5ba8-4ab0-b7d1-d9b03e60629e","Type":"ContainerDied","Data":"912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26"} Sep 16 13:23:32.928860 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:32.928813 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq"] Sep 16 13:23:32.931119 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:23:32.931087 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode041a187_a083_4a84_925b_aac987cd32db.slice/crio-1b2f226e0c45650518e6bb86b65a163a63ade4cc3f9334b65225831d9884e568 WatchSource:0}: Error finding container 1b2f226e0c45650518e6bb86b65a163a63ade4cc3f9334b65225831d9884e568: Status 404 returned error can't find the container with id 1b2f226e0c45650518e6bb86b65a163a63ade4cc3f9334b65225831d9884e568 Sep 16 13:23:33.714757 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:33.714717 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" event={"ID":"e041a187-a083-4a84-925b-aac987cd32db","Type":"ContainerStarted","Data":"a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5"} Sep 16 13:23:33.715155 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:33.714784 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" event={"ID":"e041a187-a083-4a84-925b-aac987cd32db","Type":"ContainerStarted","Data":"27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d"} Sep 16 13:23:33.715155 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:33.714814 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:33.715155 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:33.714844 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" event={"ID":"e041a187-a083-4a84-925b-aac987cd32db","Type":"ContainerStarted","Data":"1b2f226e0c45650518e6bb86b65a163a63ade4cc3f9334b65225831d9884e568"} Sep 16 13:23:33.715155 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:33.714876 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:33.716371 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:33.716345 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Sep 16 13:23:33.758836 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:33.758782 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" podStartSLOduration=2.758767516 podStartE2EDuration="2.758767516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:23:33.757472323 +0000 UTC m=+2120.950120932" watchObservedRunningTime="2026-09-16 13:23:33.758767516 +0000 UTC m=+2120.951416105" Sep 16 13:23:34.169565 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.169521 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused" Sep 16 13:23:34.173881 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.173854 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Sep 16 13:23:34.655383 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.655360 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:23:34.721514 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.721482 2584 generic.go:356] "Generic (PLEG): container finished" podID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerID="609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262" exitCode=0 Sep 16 13:23:34.721972 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.721539 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" Sep 16 13:23:34.721972 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.721568 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" event={"ID":"12d32336-5ba8-4ab0-b7d1-d9b03e60629e","Type":"ContainerDied","Data":"609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262"} Sep 16 13:23:34.721972 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.721624 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb" event={"ID":"12d32336-5ba8-4ab0-b7d1-d9b03e60629e","Type":"ContainerDied","Data":"5c58a7075f0ee7146558440bf1307d77a8915f0f70affc1b8da0aed9b8936127"} Sep 16 13:23:34.721972 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.721651 2584 scope.go:117] "RemoveContainer" containerID="912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26" Sep 16 13:23:34.722194 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.722108 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Sep 16 13:23:34.729863 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.729845 2584 scope.go:117] "RemoveContainer" containerID="609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262" Sep 16 13:23:34.737341 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.737323 2584 scope.go:117] "RemoveContainer" containerID="912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26" Sep 16 13:23:34.737636 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:23:34.737614 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26\": container with ID starting with 912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26 not found: ID does not exist" containerID="912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26" Sep 16 13:23:34.737710 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.737641 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26"} err="failed to get container status \"912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26\": rpc error: code = NotFound desc = could not find container \"912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26\": container with ID starting with 912ec40f3477fb54b45950f2194ff6cc134e6068f80f191cdc40c8d471c37b26 not found: ID does not exist" Sep 16 13:23:34.737710 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.737658 2584 scope.go:117] "RemoveContainer" containerID="609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262" Sep 16 13:23:34.737952 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:23:34.737932 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262\": container with ID starting with 609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262 not found: ID does not exist" containerID="609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262" Sep 16 13:23:34.738001 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.737960 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262"} err="failed to get container status \"609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262\": rpc error: code = NotFound desc = could not find container \"609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262\": container with ID starting with 609c840c5cb8172ac12791189b41ad34cc795ed245e2f40bbac7cb381bba6262 not found: ID does not exist" Sep 16 13:23:34.757240 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.757201 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-proxy-tls\") pod \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " Sep 16 13:23:34.757300 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.757251 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg7xm\" (UniqueName: \"kubernetes.io/projected/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-api-access-vg7xm\") pod \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " Sep 16 13:23:34.757350 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.757336 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-rbac-proxy-sar-config\") pod \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\" (UID: \"12d32336-5ba8-4ab0-b7d1-d9b03e60629e\") " Sep 16 13:23:34.757663 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.757641 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12d32336-5ba8-4ab0-b7d1-d9b03e60629e" (UID: "12d32336-5ba8-4ab0-b7d1-d9b03e60629e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:23:34.759106 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.759082 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12d32336-5ba8-4ab0-b7d1-d9b03e60629e" (UID: "12d32336-5ba8-4ab0-b7d1-d9b03e60629e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:23:34.759202 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.759153 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-api-access-vg7xm" (OuterVolumeSpecName: "kube-api-access-vg7xm") pod "12d32336-5ba8-4ab0-b7d1-d9b03e60629e" (UID: "12d32336-5ba8-4ab0-b7d1-d9b03e60629e"). InnerVolumeSpecName "kube-api-access-vg7xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:23:34.858859 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.858824 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg7xm\" (UniqueName: \"kubernetes.io/projected/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-api-access-vg7xm\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:23:34.859043 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.858963 2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:23:34.859043 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:34.858993 2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d32336-5ba8-4ab0-b7d1-d9b03e60629e-proxy-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:23:35.049789 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:35.049739 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb"] Sep 16 13:23:35.051938 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:35.051913 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d30d6-predictor-597df4949-lzkwb"] Sep 16 13:23:35.472703 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:35.472671 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" path="/var/lib/kubelet/pods/12d32336-5ba8-4ab0-b7d1-d9b03e60629e/volumes" Sep 16 13:23:39.725902 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:39.725874 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:23:39.726337 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:39.726310 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Sep 16 13:23:49.726630 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:49.726585 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Sep 16 13:23:59.726378 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:23:59.726339 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Sep 16 13:24:09.726678 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:09.726637 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Sep 16 13:24:19.728042 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:19.728011 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:24:31.516562 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.516507 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:31.618721 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.618692 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd"] Sep 16 13:24:31.618942 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.618920 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" containerID="cri-o://52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b" gracePeriod=30 Sep 16 13:24:31.619026 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.618951 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kube-rbac-proxy" containerID="cri-o://fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768" gracePeriod=30 Sep 16 13:24:31.660601 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.660569 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:31.747771 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.747741 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv"] Sep 16 13:24:31.748324 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.748306 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" Sep 16 13:24:31.748324 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.748325 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" Sep 16 13:24:31.748427 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.748347 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kube-rbac-proxy" Sep 16 13:24:31.748427 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.748356 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kube-rbac-proxy" Sep 16 13:24:31.748488 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.748439 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kserve-container" Sep 16 13:24:31.748488 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.748453 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="12d32336-5ba8-4ab0-b7d1-d9b03e60629e" containerName="kube-rbac-proxy" Sep 16 13:24:31.751669 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.751652 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:31.755114 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.755093 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a1820-kube-rbac-proxy-sar-config\"" Sep 16 13:24:31.755234 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.755138 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a1820-predictor-serving-cert\"" Sep 16 13:24:31.765360 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.765334 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv"] Sep 16 13:24:31.767679 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.767625 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:31.869415 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.869369 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:31.941488 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.941453 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-469h8\" (UniqueName: \"kubernetes.io/projected/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-api-access-469h8\") pod \"success-200-isvc-a1820-predictor-69d78bcdff-mtskv\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:31.941677 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.941509 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81a98dd5-57cb-4d56-80dc-b5322143b99b-proxy-tls\") pod \"success-200-isvc-a1820-predictor-69d78bcdff-mtskv\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:31.941677 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.941546 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1820-predictor-69d78bcdff-mtskv\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:31.945464 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.945402 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:31.985240 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.984153 2584 generic.go:356] "Generic (PLEG): container finished" podID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerID="fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768" exitCode=2 Sep 16 13:24:31.985240 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:31.984283 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" event={"ID":"a7940434-a9d6-453f-a98a-5052d19b3eb2","Type":"ContainerDied","Data":"fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768"} Sep 16 13:24:32.028695 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.028612 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:32.042972 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.042944 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-469h8\" (UniqueName: \"kubernetes.io/projected/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-api-access-469h8\") pod \"success-200-isvc-a1820-predictor-69d78bcdff-mtskv\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:32.043142 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.043001 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81a98dd5-57cb-4d56-80dc-b5322143b99b-proxy-tls\") pod \"success-200-isvc-a1820-predictor-69d78bcdff-mtskv\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:32.043142 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.043056 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1820-predictor-69d78bcdff-mtskv\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:32.043290 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:24:32.043206 2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a1820-predictor-serving-cert: secret "success-200-isvc-a1820-predictor-serving-cert" not found Sep 16 13:24:32.043290 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:24:32.043278 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a98dd5-57cb-4d56-80dc-b5322143b99b-proxy-tls podName:81a98dd5-57cb-4d56-80dc-b5322143b99b nodeName:}" failed. No retries permitted until 2026-09-16 13:24:32.543257298 +0000 UTC m=+2179.735905885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/81a98dd5-57cb-4d56-80dc-b5322143b99b-proxy-tls") pod "success-200-isvc-a1820-predictor-69d78bcdff-mtskv" (UID: "81a98dd5-57cb-4d56-80dc-b5322143b99b") : secret "success-200-isvc-a1820-predictor-serving-cert" not found Sep 16 13:24:32.043844 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.043820 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1820-predictor-69d78bcdff-mtskv\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:32.052026 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.051998 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-469h8\" (UniqueName: \"kubernetes.io/projected/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-api-access-469h8\") pod \"success-200-isvc-a1820-predictor-69d78bcdff-mtskv\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:32.377288 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.377208 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:32.478676 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.478646 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:32.546629 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.546606 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81a98dd5-57cb-4d56-80dc-b5322143b99b-proxy-tls\") pod \"success-200-isvc-a1820-predictor-69d78bcdff-mtskv\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:32.548911 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.548890 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81a98dd5-57cb-4d56-80dc-b5322143b99b-proxy-tls\") pod \"success-200-isvc-a1820-predictor-69d78bcdff-mtskv\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:32.664374 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.664311 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:32.957628 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.957595 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv"] Sep 16 13:24:32.959334 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:24:32.959310 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a98dd5_57cb_4d56_80dc_b5322143b99b.slice/crio-3c13607506044bba35e5bb16dffdd7ff229561d720739c43384d44ced41f3586 WatchSource:0}: Error finding container 3c13607506044bba35e5bb16dffdd7ff229561d720739c43384d44ced41f3586: Status 404 returned error can't find the container with id 3c13607506044bba35e5bb16dffdd7ff229561d720739c43384d44ced41f3586 Sep 16 13:24:32.989796 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:32.989772 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" event={"ID":"81a98dd5-57cb-4d56-80dc-b5322143b99b","Type":"ContainerStarted","Data":"3c13607506044bba35e5bb16dffdd7ff229561d720739c43384d44ced41f3586"} Sep 16 13:24:33.996482 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:33.996439 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" event={"ID":"81a98dd5-57cb-4d56-80dc-b5322143b99b","Type":"ContainerStarted","Data":"da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd"} Sep 16 13:24:33.996858 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:33.996513 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" event={"ID":"81a98dd5-57cb-4d56-80dc-b5322143b99b","Type":"ContainerStarted","Data":"f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13"} Sep 16 13:24:33.996858 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:33.996549 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:34.017784 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.017732 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" podStartSLOduration=3.017717198 podStartE2EDuration="3.017717198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:34.016219032 +0000 UTC m=+2181.208867640" watchObservedRunningTime="2026-09-16 13:24:34.017717198 +0000 UTC m=+2181.210365843" Sep 16 13:24:34.469528 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.469491 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused" Sep 16 13:24:34.473905 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.473876 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Sep 16 13:24:34.670998 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.670978 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:24:34.868669 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.868580 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7940434-a9d6-453f-a98a-5052d19b3eb2-proxy-tls\") pod \"a7940434-a9d6-453f-a98a-5052d19b3eb2\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " Sep 16 13:24:34.868669 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.868625 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwcdf\" (UniqueName: \"kubernetes.io/projected/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-api-access-qwcdf\") pod \"a7940434-a9d6-453f-a98a-5052d19b3eb2\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " Sep 16 13:24:34.868889 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.868704 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-rbac-proxy-sar-config\") pod \"a7940434-a9d6-453f-a98a-5052d19b3eb2\" (UID: \"a7940434-a9d6-453f-a98a-5052d19b3eb2\") " Sep 16 13:24:34.869126 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.869098 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7940434-a9d6-453f-a98a-5052d19b3eb2" (UID: "a7940434-a9d6-453f-a98a-5052d19b3eb2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:24:34.870617 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.870595 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7940434-a9d6-453f-a98a-5052d19b3eb2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7940434-a9d6-453f-a98a-5052d19b3eb2" (UID: "a7940434-a9d6-453f-a98a-5052d19b3eb2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:24:34.870676 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.870597 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-api-access-qwcdf" (OuterVolumeSpecName: "kube-api-access-qwcdf") pod "a7940434-a9d6-453f-a98a-5052d19b3eb2" (UID: "a7940434-a9d6-453f-a98a-5052d19b3eb2"). InnerVolumeSpecName "kube-api-access-qwcdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:24:34.970148 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.970117 2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7940434-a9d6-453f-a98a-5052d19b3eb2-proxy-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:24:34.970148 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.970142 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwcdf\" (UniqueName: \"kubernetes.io/projected/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-api-access-qwcdf\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:24:34.970351 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:34.970155 2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7940434-a9d6-453f-a98a-5052d19b3eb2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:24:35.003517 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.003485 2584 generic.go:356] "Generic (PLEG): container finished" podID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerID="52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b" exitCode=0 Sep 16 13:24:35.003879 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.003534 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" Sep 16 13:24:35.003879 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.003570 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" event={"ID":"a7940434-a9d6-453f-a98a-5052d19b3eb2","Type":"ContainerDied","Data":"52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b"} Sep 16 13:24:35.003879 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.003637 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd" event={"ID":"a7940434-a9d6-453f-a98a-5052d19b3eb2","Type":"ContainerDied","Data":"eb1a290929b489e7b94dbd66f71e008beae4ec323d4ed18b2c3fdb9576e90f72"} Sep 16 13:24:35.003879 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.003664 2584 scope.go:117] "RemoveContainer" containerID="fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768" Sep 16 13:24:35.004126 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.004105 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:35.005708 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.005681 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Sep 16 13:24:35.013096 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.013077 2584 scope.go:117] "RemoveContainer" containerID="52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b" Sep 16 13:24:35.020755 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.020741 2584 scope.go:117] "RemoveContainer" containerID="fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768" Sep 16 13:24:35.021001 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:24:35.020982 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768\": container with ID starting with fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768 not found: ID does not exist" containerID="fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768" Sep 16 13:24:35.021090 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.021009 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768"} err="failed to get container status \"fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768\": rpc error: code = NotFound desc = could not find container \"fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768\": container with ID starting with fec96427e355b95c75bc66984e438498d0f12660e5e4b4db06fa22aa4f34c768 not found: ID does not exist" Sep 16 13:24:35.021090 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.021024 2584 scope.go:117] "RemoveContainer" containerID="52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b" Sep 16 13:24:35.021355 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:24:35.021340 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b\": container with ID starting with 52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b not found: ID does not exist" containerID="52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b" Sep 16 13:24:35.021402 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.021359 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b"} err="failed to get container status \"52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b\": rpc error: code = NotFound desc = could not find container \"52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b\": container with ID starting with 52a1846ff516e5a2a63760f174af5816f46781ab024fefcb64d3459fa9f4a96b not found: ID does not exist" Sep 16 13:24:35.029923 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.029903 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd"] Sep 16 13:24:35.037847 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.037828 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4c3c9-predictor-58d5c86759-mspsd"] Sep 16 13:24:35.473341 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:35.473305 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" path="/var/lib/kubelet/pods/a7940434-a9d6-453f-a98a-5052d19b3eb2/volumes" Sep 16 13:24:36.009301 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:36.009261 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Sep 16 13:24:41.014110 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:41.014080 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:24:41.014640 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:41.014615 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Sep 16 13:24:41.919420 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:41.919379 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:41.979615 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:41.979582 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq"] Sep 16 13:24:41.979897 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:41.979868 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kserve-container" containerID="cri-o://27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d" gracePeriod=30 Sep 16 13:24:41.979967 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:41.979911 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kube-rbac-proxy" containerID="cri-o://a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5" gracePeriod=30 Sep 16 13:24:42.026669 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.026637 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:42.091392 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.091360 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4"] Sep 16 13:24:42.091913 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.091897 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kube-rbac-proxy" Sep 16 13:24:42.091979 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.091915 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kube-rbac-proxy" Sep 16 13:24:42.091979 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.091940 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" Sep 16 13:24:42.091979 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.091950 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" Sep 16 13:24:42.092087 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.092025 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kserve-container" Sep 16 13:24:42.092087 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.092043 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7940434-a9d6-453f-a98a-5052d19b3eb2" containerName="kube-rbac-proxy" Sep 16 13:24:42.097138 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.097116 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.100077 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.100053 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-11631-predictor-serving-cert\"" Sep 16 13:24:42.100242 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.100061 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-11631-kube-rbac-proxy-sar-config\"" Sep 16 13:24:42.108679 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.108658 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4"] Sep 16 13:24:42.120317 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.120280 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:42.123237 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.123211 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdrc\" (UniqueName: \"kubernetes.io/projected/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-api-access-jxdrc\") pod \"success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.123365 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.123342 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.123428 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.123397 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-proxy-tls\") pod \"success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.217370 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.217295 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:42.224503 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.224473 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdrc\" (UniqueName: \"kubernetes.io/projected/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-api-access-jxdrc\") pod \"success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.224657 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.224540 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.224657 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.224572 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-proxy-tls\") pod \"success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.225233 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.225212 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.226944 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.226923 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-proxy-tls\") pod \"success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.233446 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.233426 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdrc\" (UniqueName: \"kubernetes.io/projected/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-api-access-jxdrc\") pod \"success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.301616 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.301582 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:42.408524 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.408492 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:42.561622 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.561596 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:42.692371 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:42.692343 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4"] Sep 16 13:24:42.694629 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:24:42.694603 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc849f6_23f1_44d3_b685_4c3568c2b3a8.slice/crio-c29de2e4a12706c8af922b2fd2d0fdf0219569b0a24b8cf79efe9a8e2ba424f1 WatchSource:0}: Error finding container c29de2e4a12706c8af922b2fd2d0fdf0219569b0a24b8cf79efe9a8e2ba424f1: Status 404 returned error can't find the container with id c29de2e4a12706c8af922b2fd2d0fdf0219569b0a24b8cf79efe9a8e2ba424f1 Sep 16 13:24:43.040796 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:43.040711 2584 generic.go:356] "Generic (PLEG): container finished" podID="e041a187-a083-4a84-925b-aac987cd32db" containerID="a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5" exitCode=2 Sep 16 13:24:43.040796 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:43.040772 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" event={"ID":"e041a187-a083-4a84-925b-aac987cd32db","Type":"ContainerDied","Data":"a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5"} Sep 16 13:24:43.042655 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:43.042624 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" event={"ID":"3bc849f6-23f1-44d3-b685-4c3568c2b3a8","Type":"ContainerStarted","Data":"c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f"} Sep 16 13:24:43.042768 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:43.042674 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" event={"ID":"3bc849f6-23f1-44d3-b685-4c3568c2b3a8","Type":"ContainerStarted","Data":"aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09"} Sep 16 13:24:43.042768 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:43.042706 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" event={"ID":"3bc849f6-23f1-44d3-b685-4c3568c2b3a8","Type":"ContainerStarted","Data":"c29de2e4a12706c8af922b2fd2d0fdf0219569b0a24b8cf79efe9a8e2ba424f1"} Sep 16 13:24:43.042768 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:43.042745 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:43.064927 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:43.064886 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" podStartSLOduration=1.064875034 podStartE2EDuration="1.064875034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:43.063391296 +0000 UTC m=+2190.256039885" watchObservedRunningTime="2026-09-16 13:24:43.064875034 +0000 UTC m=+2190.257523623" Sep 16 13:24:44.047346 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:44.047312 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:44.048732 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:44.048699 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Sep 16 13:24:44.722721 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:44.722686 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" Sep 16 13:24:45.051530 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.051490 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Sep 16 13:24:45.225640 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.225617 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:24:45.250136 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.250070 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e041a187-a083-4a84-925b-aac987cd32db-kube-rbac-proxy-sar-config\") pod \"e041a187-a083-4a84-925b-aac987cd32db\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " Sep 16 13:24:45.250285 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.250146 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhfnv\" (UniqueName: \"kubernetes.io/projected/e041a187-a083-4a84-925b-aac987cd32db-kube-api-access-lhfnv\") pod \"e041a187-a083-4a84-925b-aac987cd32db\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " Sep 16 13:24:45.250285 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.250185 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e041a187-a083-4a84-925b-aac987cd32db-proxy-tls\") pod \"e041a187-a083-4a84-925b-aac987cd32db\" (UID: \"e041a187-a083-4a84-925b-aac987cd32db\") " Sep 16 13:24:45.250532 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.250504 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e041a187-a083-4a84-925b-aac987cd32db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e041a187-a083-4a84-925b-aac987cd32db" (UID: "e041a187-a083-4a84-925b-aac987cd32db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:24:45.252225 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.252203 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e041a187-a083-4a84-925b-aac987cd32db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e041a187-a083-4a84-925b-aac987cd32db" (UID: "e041a187-a083-4a84-925b-aac987cd32db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:24:45.252656 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.252636 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e041a187-a083-4a84-925b-aac987cd32db-kube-api-access-lhfnv" (OuterVolumeSpecName: "kube-api-access-lhfnv") pod "e041a187-a083-4a84-925b-aac987cd32db" (UID: "e041a187-a083-4a84-925b-aac987cd32db"). InnerVolumeSpecName "kube-api-access-lhfnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:24:45.351524 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.351493 2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e041a187-a083-4a84-925b-aac987cd32db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:24:45.351524 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.351519 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhfnv\" (UniqueName: \"kubernetes.io/projected/e041a187-a083-4a84-925b-aac987cd32db-kube-api-access-lhfnv\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:24:45.351524 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:45.351528 2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e041a187-a083-4a84-925b-aac987cd32db-proxy-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:24:46.058391 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.058358 2584 generic.go:356] "Generic (PLEG): container finished" podID="e041a187-a083-4a84-925b-aac987cd32db" containerID="27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d" exitCode=0 Sep 16 13:24:46.058831 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.058409 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" Sep 16 13:24:46.058831 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.058444 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" event={"ID":"e041a187-a083-4a84-925b-aac987cd32db","Type":"ContainerDied","Data":"27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d"} Sep 16 13:24:46.058831 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.058506 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq" event={"ID":"e041a187-a083-4a84-925b-aac987cd32db","Type":"ContainerDied","Data":"1b2f226e0c45650518e6bb86b65a163a63ade4cc3f9334b65225831d9884e568"} Sep 16 13:24:46.058831 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.058533 2584 scope.go:117] "RemoveContainer" containerID="a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5" Sep 16 13:24:46.066799 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.066781 2584 scope.go:117] "RemoveContainer" containerID="27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d" Sep 16 13:24:46.074093 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.074078 2584 scope.go:117] "RemoveContainer" containerID="a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5" Sep 16 13:24:46.074383 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:24:46.074365 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5\": container with ID starting with a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5 not found: ID does not exist" containerID="a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5" Sep 16 13:24:46.074427 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.074388 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5"} err="failed to get container status \"a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5\": rpc error: code = NotFound desc = could not find container \"a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5\": container with ID starting with a2e30caf0fcfabe6b6eac3dbcc015ab90f162eaa7232ecfd4602978405fc50a5 not found: ID does not exist" Sep 16 13:24:46.074427 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.074403 2584 scope.go:117] "RemoveContainer" containerID="27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d" Sep 16 13:24:46.074638 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:24:46.074624 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d\": container with ID starting with 27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d not found: ID does not exist" containerID="27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d" Sep 16 13:24:46.074691 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.074642 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d"} err="failed to get container status \"27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d\": rpc error: code = NotFound desc = could not find container \"27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d\": container with ID starting with 27a1f1b90b8cc99db8f13a3638241d94150b2de7d0d7038ba226c3a76b0b9c5d not found: ID does not exist" Sep 16 13:24:46.088916 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.088893 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq"] Sep 16 13:24:46.094567 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:46.094542 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e1666-predictor-6fcf567cf-z5ktq"] Sep 16 13:24:47.472928 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:47.472895 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e041a187-a083-4a84-925b-aac987cd32db" path="/var/lib/kubelet/pods/e041a187-a083-4a84-925b-aac987cd32db/volumes" Sep 16 13:24:50.056389 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:50.056355 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:24:50.057008 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:50.056977 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Sep 16 13:24:51.015233 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:24:51.015193 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Sep 16 13:25:00.057025 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:00.056985 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Sep 16 13:25:01.015117 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:01.015082 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Sep 16 13:25:10.057217 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:10.057151 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Sep 16 13:25:11.014860 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:11.014823 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Sep 16 13:25:20.057088 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:20.057047 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Sep 16 13:25:21.015824 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:21.015790 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:25:30.058928 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:30.058897 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:25:31.787145 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:31.787109 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:32.183542 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:32.183457 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:42.172242 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:42.172201 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:42.817255 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:25:42.817220 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:27.752308 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:26:27.752274 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:28:13.465677 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:28:13.465577 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:28:13.469620 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:28:13.469243 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:28:13.474929 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:28:13.474911 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:28:13.477864 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:28:13.477847 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:29:54.672985 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:29:54.672948 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:38.497834 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:30:38.497798 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:59.063401 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:30:59.063365 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:13.498370 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:13.498341 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:33:13.503673 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:13.503653 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:33:13.506584 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:13.506560 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:33:13.511521 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:13.511507 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:33:56.380387 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:56.380351 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:56.832658 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:56.832622 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4"] Sep 16 13:33:56.832972 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:56.832941 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kserve-container" containerID="cri-o://aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09" gracePeriod=30 Sep 16 13:33:56.833048 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:56.832958 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kube-rbac-proxy" containerID="cri-o://c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f" gracePeriod=30 Sep 16 13:33:57.538745 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:57.538712 2584 generic.go:356] "Generic (PLEG): container finished" podID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerID="c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f" exitCode=2 Sep 16 13:33:57.539130 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:57.538762 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" event={"ID":"3bc849f6-23f1-44d3-b685-4c3568c2b3a8","Type":"ContainerDied","Data":"c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f"} Sep 16 13:33:58.836503 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:58.836443 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:59.876003 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:33:59.875977 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:34:00.012153 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.012106 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxdrc\" (UniqueName: \"kubernetes.io/projected/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-api-access-jxdrc\") pod \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " Sep 16 13:34:00.012357 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.012199 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-proxy-tls\") pod \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " Sep 16 13:34:00.012357 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.012266 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-rbac-proxy-sar-config\") pod \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\" (UID: \"3bc849f6-23f1-44d3-b685-4c3568c2b3a8\") " Sep 16 13:34:00.012641 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.012618 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3bc849f6-23f1-44d3-b685-4c3568c2b3a8" (UID: "3bc849f6-23f1-44d3-b685-4c3568c2b3a8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:34:00.014292 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.014265 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-api-access-jxdrc" (OuterVolumeSpecName: "kube-api-access-jxdrc") pod "3bc849f6-23f1-44d3-b685-4c3568c2b3a8" (UID: "3bc849f6-23f1-44d3-b685-4c3568c2b3a8"). InnerVolumeSpecName "kube-api-access-jxdrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:34:00.014394 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.014288 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bc849f6-23f1-44d3-b685-4c3568c2b3a8" (UID: "3bc849f6-23f1-44d3-b685-4c3568c2b3a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:34:00.113514 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.113483 2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:34:00.113514 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.113512 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdrc\" (UniqueName: \"kubernetes.io/projected/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-kube-api-access-jxdrc\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:34:00.113695 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.113522 2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bc849f6-23f1-44d3-b685-4c3568c2b3a8-proxy-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:34:00.552976 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.552944 2584 generic.go:356] "Generic (PLEG): container finished" podID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerID="aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09" exitCode=0 Sep 16 13:34:00.553122 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.552989 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" Sep 16 13:34:00.553122 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.553026 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" event={"ID":"3bc849f6-23f1-44d3-b685-4c3568c2b3a8","Type":"ContainerDied","Data":"aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09"} Sep 16 13:34:00.553122 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.553085 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4" event={"ID":"3bc849f6-23f1-44d3-b685-4c3568c2b3a8","Type":"ContainerDied","Data":"c29de2e4a12706c8af922b2fd2d0fdf0219569b0a24b8cf79efe9a8e2ba424f1"} Sep 16 13:34:00.553122 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.553112 2584 scope.go:117] "RemoveContainer" containerID="c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f" Sep 16 13:34:00.561372 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.561358 2584 scope.go:117] "RemoveContainer" containerID="aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09" Sep 16 13:34:00.568940 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.568922 2584 scope.go:117] "RemoveContainer" containerID="c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f" Sep 16 13:34:00.569259 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:34:00.569236 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f\": container with ID starting with c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f not found: ID does not exist" containerID="c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f" Sep 16 13:34:00.569317 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.569269 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f"} err="failed to get container status \"c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f\": rpc error: code = NotFound desc = could not find container \"c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f\": container with ID starting with c51cd925bb40918834b7b14d111aa71b9821dfff1c6a9fb2b9e5396b351ea17f not found: ID does not exist" Sep 16 13:34:00.569317 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.569285 2584 scope.go:117] "RemoveContainer" containerID="aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09" Sep 16 13:34:00.569564 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:34:00.569542 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09\": container with ID starting with aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09 not found: ID does not exist" containerID="aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09" Sep 16 13:34:00.569624 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.569570 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09"} err="failed to get container status \"aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09\": rpc error: code = NotFound desc = could not find container \"aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09\": container with ID starting with aecdda13dc441c97be1bec6570da520a2888d0acf65c0ed4ec711d8366351a09 not found: ID does not exist" Sep 16 13:34:00.575225 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.575204 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4"] Sep 16 13:34:00.578742 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:00.578720 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11631-predictor-57bc9c8bc9-gcmh4"] Sep 16 13:34:01.473334 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:34:01.473291 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" path="/var/lib/kubelet/pods/3bc849f6-23f1-44d3-b685-4c3568c2b3a8/volumes" Sep 16 13:36:58.648733 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:36:58.648698 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:13.528774 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:38:13.528678 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:38:13.534097 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:38:13.534076 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:38:13.536750 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:38:13.536732 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:38:13.541688 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:38:13.541671 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:39:00.469499 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:39:00.469447 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:50.959412 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:50.959380 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:51.026226 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:51.026193 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv"] Sep 16 13:41:51.026528 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:51.026479 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kserve-container" containerID="cri-o://f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13" gracePeriod=30 Sep 16 13:41:51.026665 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:51.026518 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kube-rbac-proxy" containerID="cri-o://da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd" gracePeriod=30 Sep 16 13:41:51.049297 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:51.049243 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:51.112037 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:51.112008 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:51.636036 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:51.636005 2584 generic.go:356] "Generic (PLEG): container finished" podID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerID="da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd" exitCode=2 Sep 16 13:41:51.636243 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:51.636074 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" event={"ID":"81a98dd5-57cb-4d56-80dc-b5322143b99b","Type":"ContainerDied","Data":"da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd"} Sep 16 13:41:52.295285 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:52.295243 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:52.395967 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:52.395934 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:53.975477 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:53.975447 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:41:54.120039 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.119964 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-rbac-proxy-sar-config\") pod \"81a98dd5-57cb-4d56-80dc-b5322143b99b\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " Sep 16 13:41:54.120039 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.120030 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81a98dd5-57cb-4d56-80dc-b5322143b99b-proxy-tls\") pod \"81a98dd5-57cb-4d56-80dc-b5322143b99b\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " Sep 16 13:41:54.120300 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.120068 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-469h8\" (UniqueName: \"kubernetes.io/projected/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-api-access-469h8\") pod \"81a98dd5-57cb-4d56-80dc-b5322143b99b\" (UID: \"81a98dd5-57cb-4d56-80dc-b5322143b99b\") " Sep 16 13:41:54.120403 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.120380 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81a98dd5-57cb-4d56-80dc-b5322143b99b" (UID: "81a98dd5-57cb-4d56-80dc-b5322143b99b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:41:54.122117 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.122084 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a98dd5-57cb-4d56-80dc-b5322143b99b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81a98dd5-57cb-4d56-80dc-b5322143b99b" (UID: "81a98dd5-57cb-4d56-80dc-b5322143b99b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:41:54.122242 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.122115 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-api-access-469h8" (OuterVolumeSpecName: "kube-api-access-469h8") pod "81a98dd5-57cb-4d56-80dc-b5322143b99b" (UID: "81a98dd5-57cb-4d56-80dc-b5322143b99b"). InnerVolumeSpecName "kube-api-access-469h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:41:54.221006 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.220982 2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81a98dd5-57cb-4d56-80dc-b5322143b99b-proxy-tls\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:41:54.221006 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.221003 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-469h8\" (UniqueName: \"kubernetes.io/projected/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-api-access-469h8\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:41:54.221145 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.221014 2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81a98dd5-57cb-4d56-80dc-b5322143b99b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-139.ec2.internal\" DevicePath \"\"" Sep 16 13:41:54.651305 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.651272 2584 generic.go:356] "Generic (PLEG): container finished" podID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerID="f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13" exitCode=0 Sep 16 13:41:54.651305 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.651310 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" event={"ID":"81a98dd5-57cb-4d56-80dc-b5322143b99b","Type":"ContainerDied","Data":"f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13"} Sep 16 13:41:54.651536 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.651336 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" Sep 16 13:41:54.651536 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.651357 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv" event={"ID":"81a98dd5-57cb-4d56-80dc-b5322143b99b","Type":"ContainerDied","Data":"3c13607506044bba35e5bb16dffdd7ff229561d720739c43384d44ced41f3586"} Sep 16 13:41:54.651536 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.651384 2584 scope.go:117] "RemoveContainer" containerID="da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd" Sep 16 13:41:54.660137 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.660117 2584 scope.go:117] "RemoveContainer" containerID="f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13" Sep 16 13:41:54.667786 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.667766 2584 scope.go:117] "RemoveContainer" containerID="da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd" Sep 16 13:41:54.668060 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:41:54.668038 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd\": container with ID starting with da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd not found: ID does not exist" containerID="da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd" Sep 16 13:41:54.668121 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.668069 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd"} err="failed to get container status \"da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd\": rpc error: code = NotFound desc = could not find container \"da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd\": container with ID starting with da4bf692b167531048ff432aa73d9cf8cc24487277b4497fb60beec06f5992fd not found: ID does not exist" Sep 16 13:41:54.668121 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.668084 2584 scope.go:117] "RemoveContainer" containerID="f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13" Sep 16 13:41:54.668778 ip-10-0-141-139 kubenswrapper[2584]: E0916 13:41:54.668761 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13\": container with ID starting with f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13 not found: ID does not exist" containerID="f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13" Sep 16 13:41:54.668826 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.668785 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13"} err="failed to get container status \"f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13\": rpc error: code = NotFound desc = could not find container \"f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13\": container with ID starting with f012b0769d7b4f5c82a1ded3db0cc4d339cddc6dbf2b2acb3748ee49c8697a13 not found: ID does not exist" Sep 16 13:41:54.676678 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.676655 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv"] Sep 16 13:41:54.679142 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:54.679121 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1820-predictor-69d78bcdff-mtskv"] Sep 16 13:41:55.477984 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:41:55.475599 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" path="/var/lib/kubelet/pods/81a98dd5-57cb-4d56-80dc-b5322143b99b/volumes" Sep 16 13:42:23.525990 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:23.525932 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:23.683458 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:23.683426 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:23.740514 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:23.740480 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:26.330572 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:26.330536 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrjnz_00648523-fd55-498f-b8c6-704f6c9d7e47/global-pull-secret-syncer/0.log" Sep 16 13:42:26.422216 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:26.422186 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" Sep 16 13:42:26.560074 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:26.560047 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-139.ec2.internal_8b76f1b2a16574ce38927f7f5da97679/haproxy/0.log" Sep 16 13:42:29.226607 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:29.226576 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:29.289492 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:29.289456 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fgbqm_37248adf-9ac8-4fd6-a693-daf752b60555/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:30.490310 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:30.490282 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1d7c3b9-66ea-4d17-a0ed-e95a147502b3/alertmanager/0.log" Sep 16 13:42:30.531537 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:30.531511 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1d7c3b9-66ea-4d17-a0ed-e95a147502b3/config-reloader/0.log" Sep 16 13:42:30.561537 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:30.561511 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1d7c3b9-66ea-4d17-a0ed-e95a147502b3/kube-rbac-proxy-web/0.log" Sep 16 13:42:30.589425 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:30.589398 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1d7c3b9-66ea-4d17-a0ed-e95a147502b3/kube-rbac-proxy/0.log" Sep 16 13:42:30.618410 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:30.618382 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1d7c3b9-66ea-4d17-a0ed-e95a147502b3/kube-rbac-proxy-metric/0.log" Sep 16 13:42:30.641272 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:30.641245 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1d7c3b9-66ea-4d17-a0ed-e95a147502b3/prom-label-proxy/0.log" Sep 16 13:42:30.664017 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:30.663993 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c1d7c3b9-66ea-4d17-a0ed-e95a147502b3/init-config-reloader/0.log" Sep 16 13:42:30.739213 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:30.739179 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cns4d_d11aa326-cac9-419e-9382-7e34a43fd46b/kube-state-metrics/0.log" Sep 16 13:42:30.759511 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:30.759450 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cns4d_d11aa326-cac9-419e-9382-7e34a43fd46b/kube-rbac-proxy-main/0.log" Sep 16 13:42:30.782007 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:30.781984 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cns4d_d11aa326-cac9-419e-9382-7e34a43fd46b/kube-rbac-proxy-self/0.log" Sep 16 13:42:31.104566 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:31.104489 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m9qss_59664ff4-a68b-41fa-b0ee-4b7f85aa41da/node-exporter/0.log" Sep 16 13:42:31.130819 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:31.130795 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m9qss_59664ff4-a68b-41fa-b0ee-4b7f85aa41da/kube-rbac-proxy/0.log" Sep 16 13:42:31.155775 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:31.155750 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m9qss_59664ff4-a68b-41fa-b0ee-4b7f85aa41da/init-textfile/0.log" Sep 16 13:42:31.541003 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:31.540969 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-g5qkk_661e33e1-a996-4039-9f50-3a7e4b16ebd7/prometheus-operator-admission-webhook/0.log" Sep 16 13:42:31.583271 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:31.583242 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f67954898-rtjkd_e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38/telemeter-client/0.log" Sep 16 13:42:31.605448 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:31.605405 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f67954898-rtjkd_e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38/reload/0.log" Sep 16 13:42:31.629014 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:31.628986 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f67954898-rtjkd_e71ebd8f-bb3f-4e29-9a06-8d4312c6eb38/kube-rbac-proxy/0.log" Sep 16 13:42:33.471411 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:33.471375 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/1.log" Sep 16 13:42:33.476115 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:33.476089 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8stkv_ccb21282-c2dd-405b-9041-83c4ec35c7a3/console-operator/2.log" Sep 16 13:42:34.501871 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.501833 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn"] Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502152 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kube-rbac-proxy" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502178 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kube-rbac-proxy" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502187 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kserve-container" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502193 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kserve-container" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502202 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kube-rbac-proxy" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502208 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kube-rbac-proxy" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502215 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kserve-container" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502220 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kserve-container" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502226 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kube-rbac-proxy" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502232 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kube-rbac-proxy" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502240 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kserve-container" Sep 16 13:42:34.502273 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502245 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kserve-container" Sep 16 13:42:34.502647 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502304 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kserve-container" Sep 16 13:42:34.502647 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502313 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kube-rbac-proxy" Sep 16 13:42:34.502647 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502319 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="81a98dd5-57cb-4d56-80dc-b5322143b99b" containerName="kube-rbac-proxy" Sep 16 13:42:34.502647 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502326 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kserve-container" Sep 16 13:42:34.502647 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502330 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e041a187-a083-4a84-925b-aac987cd32db" containerName="kube-rbac-proxy" Sep 16 13:42:34.502647 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.502337 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc849f6-23f1-44d3-b685-4c3568c2b3a8" containerName="kserve-container" Sep 16 13:42:34.505389 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.505362 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.508009 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.507987 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-svrn2\"/\"default-dockercfg-prqsv\"" Sep 16 13:42:34.509263 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.509241 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-svrn2\"/\"kube-root-ca.crt\"" Sep 16 13:42:34.509368 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.509324 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-svrn2\"/\"openshift-service-ca.crt\"" Sep 16 13:42:34.514657 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.514634 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn"] Sep 16 13:42:34.536448 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.536418 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-proc\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.536575 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.536461 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp6gt\" (UniqueName: \"kubernetes.io/projected/a59224da-767c-4339-9ea8-81496862f8e3-kube-api-access-pp6gt\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.536575 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.536509 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-podres\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.536575 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.536553 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-sys\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.536684 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.536586 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-lib-modules\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.637806 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.637771 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-podres\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.637806 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.637818 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-sys\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.638054 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.637840 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-lib-modules\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.638054 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.637886 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-proc\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.638054 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.637911 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-sys\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.638054 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.637917 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp6gt\" (UniqueName: \"kubernetes.io/projected/a59224da-767c-4339-9ea8-81496862f8e3-kube-api-access-pp6gt\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.638054 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.637960 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-podres\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.638054 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.637973 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-proc\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.638054 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.638018 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a59224da-767c-4339-9ea8-81496862f8e3-lib-modules\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.646337 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.646320 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp6gt\" (UniqueName: \"kubernetes.io/projected/a59224da-767c-4339-9ea8-81496862f8e3-kube-api-access-pp6gt\") pod \"perf-node-gather-daemonset-lkdwn\" (UID: \"a59224da-767c-4339-9ea8-81496862f8e3\") " pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:34.815901 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:34.815806 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:35.090134 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:35.090070 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn"] Sep 16 13:42:35.092572 ip-10-0-141-139 kubenswrapper[2584]: W0916 13:42:35.092541 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda59224da_767c_4339_9ea8_81496862f8e3.slice/crio-83ccd038d891ece0a8a6be4c471982e8efbbb504ef0a0a08944662d79957f93e WatchSource:0}: Error finding container 83ccd038d891ece0a8a6be4c471982e8efbbb504ef0a0a08944662d79957f93e: Status 404 returned error can't find the container with id 83ccd038d891ece0a8a6be4c471982e8efbbb504ef0a0a08944662d79957f93e Sep 16 13:42:35.408415 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:35.408336 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vvb26_42ae37fd-a147-4072-b59d-ebbda3c45206/dns/0.log" Sep 16 13:42:35.424594 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:35.424568 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vvb26_42ae37fd-a147-4072-b59d-ebbda3c45206/kube-rbac-proxy/0.log" Sep 16 13:42:35.444861 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:35.444831 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c64wg_9711ead1-9753-4be1-b0f5-78cd0ac6f89f/dns-node-resolver/0.log" Sep 16 13:42:35.828834 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:35.828796 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" event={"ID":"a59224da-767c-4339-9ea8-81496862f8e3","Type":"ContainerStarted","Data":"faa2ea50538dd71cb1fcfd0633b9f6ef00d3b0ae82efd92af2ba24e4cb26720d"} Sep 16 13:42:35.829257 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:35.828859 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" event={"ID":"a59224da-767c-4339-9ea8-81496862f8e3","Type":"ContainerStarted","Data":"83ccd038d891ece0a8a6be4c471982e8efbbb504ef0a0a08944662d79957f93e"} Sep 16 13:42:35.829257 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:35.828887 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:35.848843 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:35.848801 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" podStartSLOduration=1.848784024 podStartE2EDuration="1.848784024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:42:35.848538342 +0000 UTC m=+3263.041186952" watchObservedRunningTime="2026-09-16 13:42:35.848784024 +0000 UTC m=+3263.041432614" Sep 16 13:42:35.960372 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:35.960343 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gbhf2_b6555a32-a5f6-4d65-82c7-6aaabaff1fbc/node-ca/0.log" Sep 16 13:42:37.096735 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:37.096708 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-st6jz_49a4be54-687a-4031-9487-72e8d73bdc3d/serve-healthcheck-canary/0.log" Sep 16 13:42:37.498501 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:37.498475 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jmh42_8f1b53d5-d2b5-437e-8944-96aa1dd90fce/kube-rbac-proxy/0.log" Sep 16 13:42:37.516010 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:37.515985 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jmh42_8f1b53d5-d2b5-437e-8944-96aa1dd90fce/exporter/0.log" Sep 16 13:42:37.533594 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:37.533576 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jmh42_8f1b53d5-d2b5-437e-8944-96aa1dd90fce/extractor/0.log" Sep 16 13:42:39.811220 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:39.811191 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-867644d759-8dqsv_5dfcabbe-dc77-4541-93d9-8ec5b8d132fc/manager/0.log" Sep 16 13:42:40.072619 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:40.072543 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-rj5f2_533b15c8-d8dd-4b33-bb95-769fae375156/seaweedfs/0.log" Sep 16 13:42:41.842823 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:41.842797 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-svrn2/perf-node-gather-daemonset-lkdwn" Sep 16 13:42:43.956879 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:43.956849 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-v5nz2_a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16/migrator/0.log" Sep 16 13:42:43.981860 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:43.981793 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-v5nz2_a97a6fd4-8eea-44e9-b4d6-fc5b7c18ed16/graceful-termination/0.log" Sep 16 13:42:44.391105 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:44.391029 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-vd2fc_d0dc2888-aafe-47b0-bf94-0ba5fe2238c5/kube-storage-version-migrator-operator/1.log" Sep 16 13:42:44.391974 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:44.391952 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-vd2fc_d0dc2888-aafe-47b0-bf94-0ba5fe2238c5/kube-storage-version-migrator-operator/0.log" Sep 16 13:42:45.235621 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:45.235572 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bslcs_be40f43f-3d2a-4499-8e88-b78cd35bc9ed/kube-multus-additional-cni-plugins/0.log" Sep 16 13:42:45.258041 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:45.258015 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bslcs_be40f43f-3d2a-4499-8e88-b78cd35bc9ed/egress-router-binary-copy/0.log" Sep 16 13:42:45.277960 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:45.277936 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bslcs_be40f43f-3d2a-4499-8e88-b78cd35bc9ed/cni-plugins/0.log" Sep 16 13:42:45.297993 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:45.297971 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bslcs_be40f43f-3d2a-4499-8e88-b78cd35bc9ed/bond-cni-plugin/0.log" Sep 16 13:42:45.316462 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:45.316442 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bslcs_be40f43f-3d2a-4499-8e88-b78cd35bc9ed/routeoverride-cni/0.log" Sep 16 13:42:45.334247 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:45.334228 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bslcs_be40f43f-3d2a-4499-8e88-b78cd35bc9ed/whereabouts-cni-bincopy/0.log" Sep 16 13:42:45.352690 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:45.352667 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bslcs_be40f43f-3d2a-4499-8e88-b78cd35bc9ed/whereabouts-cni/0.log" Sep 16 13:42:45.703640 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:45.703553 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g7gz4_b868f668-a89e-43dd-bb29-18c856372b90/kube-multus/0.log" Sep 16 13:42:45.822979 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:45.822948 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jjdcd_418a1352-4779-435a-880c-627e3480b111/network-metrics-daemon/0.log" Sep 16 13:42:45.837861 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:45.837836 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jjdcd_418a1352-4779-435a-880c-627e3480b111/kube-rbac-proxy/0.log" Sep 16 13:42:46.937672 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:46.937637 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-controller/0.log" Sep 16 13:42:46.961486 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:46.961462 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/1.log" Sep 16 13:42:46.961486 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:46.961483 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovn-acl-logging/0.log" Sep 16 13:42:46.977473 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:46.977450 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/kube-rbac-proxy-node/0.log" Sep 16 13:42:46.998306 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:46.998284 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 13:42:47.016543 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:47.016519 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/northd/0.log" Sep 16 13:42:47.040323 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:47.040300 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/nbdb/0.log" Sep 16 13:42:47.065697 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:47.065671 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/sbdb/0.log" Sep 16 13:42:47.109605 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:47.109578 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzmzs_10c6c3a7-6ac2-45ea-9726-28260b38b6d8/ovnkube-controller/0.log" Sep 16 13:42:48.146335 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:48.146308 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-npbm5_3cced485-3b23-4c6b-acc3-4d644b0ff0ae/check-endpoints/0.log" Sep 16 13:42:48.170865 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:48.170839 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-btfcd_aacaaf0a-8a6f-4ced-9c88-1fab1496cb04/network-check-target-container/0.log" Sep 16 13:42:49.061785 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:49.061758 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2259d_4f90e3ec-14f4-443f-b93a-1a1d05265a71/iptables-alerter/0.log" Sep 16 13:42:49.664998 ip-10-0-141-139 kubenswrapper[2584]: I0916 13:42:49.664966 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9zltf_dda716d8-6057-42d2-8a6f-ee5cddd8592a/tuned/0.log"