Sep 18 16:13:36.247610 ip-10-0-143-79 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 18 16:13:36.247623 ip-10-0-143-79 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 18 16:13:36.247630 ip-10-0-143-79 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 18 16:13:36.247778 ip-10-0-143-79 systemd[1]: Failed to start Kubernetes Kubelet. Sep 18 16:13:46.459287 ip-10-0-143-79 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 18 16:13:46.459305 ip-10-0-143-79 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d2786a4fdd3c4245bee7f60efc13f286 -- Sep 18 16:15:31.860750 ip-10-0-143-79 systemd[1]: Starting Kubernetes Kubelet... Sep 18 16:15:35.418094 ip-10-0-143-79 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:15:35.418094 ip-10-0-143-79 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 18 16:15:35.418094 ip-10-0-143-79 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:15:35.418094 ip-10-0-143-79 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 18 16:15:35.418094 ip-10-0-143-79 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:15:35.418897 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.418167 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 18 16:15:35.421047 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421030 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:15:35.421047 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421047 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421052 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421057 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421061 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421064 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421067 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421070 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421073 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421076 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421079 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421081 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421084 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421086 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421090 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421092 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421095 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421097 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421100 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421102 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421105 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:15:35.421105 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421108 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421110 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421113 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421116 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421119 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421123 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421125 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421128 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421130 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421133 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421135 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421138 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421140 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421143 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421145 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421148 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421150 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421153 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421155 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421157 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:15:35.421579 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421160 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421162 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421164 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421167 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421170 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421172 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421175 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421177 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421179 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421182 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421184 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421187 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421190 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421193 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421196 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421199 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421201 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421205 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421209 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:15:35.422090 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421212 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421215 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421218 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421221 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421224 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421227 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421230 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421232 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421235 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421238 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421241 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421243 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421246 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421248 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421251 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421253 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421257 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421261 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421264 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421266 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:15:35.422549 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421269 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421271 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421275 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421278 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421280 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421283 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421285 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421289 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421292 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421295 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421297 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421300 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421303 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421305 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421308 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421310 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421313 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421315 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421318 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421980 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:15:35.423025 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421986 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421989 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421992 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421996 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.421999 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422001 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422004 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422006 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422009 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422011 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422014 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422017 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422020 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422023 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422025 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422028 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422031 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422033 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422036 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422039 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:15:35.423514 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422041 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422043 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422046 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422048 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422051 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422054 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422056 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422059 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422061 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422064 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422066 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422069 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422071 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422074 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422077 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422080 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422082 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422085 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422087 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422090 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:15:35.423996 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422093 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422096 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422099 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422102 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422104 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422107 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422109 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422111 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422114 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422117 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422119 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422122 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422125 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422127 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422129 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422132 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422134 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422137 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422141 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422145 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:15:35.424478 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422148 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422151 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422154 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422158 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422161 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422164 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422167 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422169 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422172 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422175 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422177 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422180 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422183 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422186 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422189 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422191 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422194 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422197 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422199 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:15:35.424978 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422201 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422204 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422206 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422209 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422212 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422256 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422261 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422265 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422267 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422270 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422273 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422276 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422279 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422281 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422284 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422287 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422290 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422293 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422296 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:15:35.425421 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422695 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422702 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422705 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422708 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422711 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422714 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422716 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422719 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422721 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422724 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422727 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422729 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422732 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422734 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422737 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422740 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422743 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422746 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422748 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422751 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:15:35.425891 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422753 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422757 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422759 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422762 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422764 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422767 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422769 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422773 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422776 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422780 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422783 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422785 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422789 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422791 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422794 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422797 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422799 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422802 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422804 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422807 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:15:35.426363 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422810 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422812 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422814 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422817 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422819 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422822 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422824 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422826 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422829 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422831 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422834 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422837 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422839 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422843 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422846 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422849 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422852 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422855 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422858 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:15:35.426903 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422860 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422863 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422865 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422868 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422871 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422874 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422876 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422879 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422882 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422885 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422887 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422889 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422892 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422894 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422897 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422899 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422902 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422904 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422907 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422910 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:15:35.427383 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422913 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422916 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422919 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422922 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422925 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422928 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422931 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422933 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422936 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422938 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422941 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422943 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422945 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422948 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422951 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422954 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422956 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422958 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422961 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.422963 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423034 2564 flags.go:64] FLAG: --address="0.0.0.0" Sep 18 16:15:35.427864 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423081 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423088 2564 flags.go:64] FLAG: --anonymous-auth="true" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423092 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423096 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423099 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423103 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423107 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423110 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423113 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423116 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423120 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423123 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423126 2564 flags.go:64] FLAG: --cgroup-root="" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423129 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423132 2564 flags.go:64] FLAG: --client-ca-file="" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423135 2564 flags.go:64] FLAG: --cloud-provider="external" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423138 2564 flags.go:64] FLAG: --cluster-dns="[]" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423145 2564 flags.go:64] FLAG: --cluster-domain="" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423148 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423151 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423154 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423158 2564 flags.go:64] FLAG: --container-log-max-files="5" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423161 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 18 16:15:35.428370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423164 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423168 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423173 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423176 2564 flags.go:64] FLAG: --contention-profiling="false" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423179 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423182 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423185 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423188 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423193 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423196 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423199 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423202 2564 flags.go:64] FLAG: --enable-load-reader="false" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423206 2564 flags.go:64] FLAG: --enable-server="true" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423209 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423213 2564 flags.go:64] FLAG: --event-burst="100" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423216 2564 flags.go:64] FLAG: --event-qps="50" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423219 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423222 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423225 2564 flags.go:64] FLAG: --eviction-hard="" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423228 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423231 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423234 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423239 2564 flags.go:64] FLAG: --eviction-soft="" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423242 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423244 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 18 16:15:35.428938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423247 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423250 2564 flags.go:64] FLAG: --experimental-mounter-path="" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423253 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423256 2564 flags.go:64] FLAG: --fail-swap-on="true" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423259 2564 flags.go:64] FLAG: --feature-gates="" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423263 2564 flags.go:64] FLAG: --file-check-frequency="20s" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423266 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423269 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423272 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423277 2564 flags.go:64] FLAG: --healthz-port="10248" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423280 2564 flags.go:64] FLAG: --help="false" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423283 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423286 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423289 2564 flags.go:64] FLAG: --http-check-frequency="20s" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423292 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423295 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423299 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423301 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423304 2564 flags.go:64] FLAG: --image-service-endpoint="" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423308 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423311 2564 flags.go:64] FLAG: --kube-api-burst="100" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423313 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423317 2564 flags.go:64] FLAG: --kube-api-qps="50" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423319 2564 flags.go:64] FLAG: --kube-reserved="" Sep 18 16:15:35.429547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423322 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423325 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423328 2564 flags.go:64] FLAG: --kubelet-cgroups="" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423331 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423334 2564 flags.go:64] FLAG: --lock-file="" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423338 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423341 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423343 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423349 2564 flags.go:64] FLAG: --log-json-split-stream="false" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423351 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423354 2564 flags.go:64] FLAG: --log-text-split-stream="false" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423357 2564 flags.go:64] FLAG: --logging-format="text" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423360 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423363 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423365 2564 flags.go:64] FLAG: --manifest-url="" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423368 2564 flags.go:64] FLAG: --manifest-url-header="" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423373 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423379 2564 flags.go:64] FLAG: --max-open-files="1000000" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423382 2564 flags.go:64] FLAG: --max-pods="110" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423385 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423388 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423391 2564 flags.go:64] FLAG: --memory-manager-policy="None" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423394 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423397 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423399 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 18 16:15:35.430205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423402 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423409 2564 flags.go:64] FLAG: --node-status-max-images="50" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423412 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423415 2564 flags.go:64] FLAG: --oom-score-adj="-999" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423418 2564 flags.go:64] FLAG: --pod-cidr="" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423421 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423426 2564 flags.go:64] FLAG: --pod-manifest-path="" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423429 2564 flags.go:64] FLAG: --pod-max-pids="-1" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423432 2564 flags.go:64] FLAG: --pods-per-core="0" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423436 2564 flags.go:64] FLAG: --port="10250" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423439 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423442 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ec03cc32a04607b0" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423446 2564 flags.go:64] FLAG: --qos-reserved="" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423449 2564 flags.go:64] FLAG: --read-only-port="10255" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423451 2564 flags.go:64] FLAG: --register-node="true" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423454 2564 flags.go:64] FLAG: --register-with-taints="" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423458 2564 flags.go:64] FLAG: --registry-burst="10" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423460 2564 flags.go:64] FLAG: --registry-qps="5" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423463 2564 flags.go:64] FLAG: --reserved-cpus="" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423466 2564 flags.go:64] FLAG: --reserved-memory="" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423470 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423473 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423476 2564 flags.go:64] FLAG: --rotate-certificates="false" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423478 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423494 2564 flags.go:64] FLAG: --runonce="false" Sep 18 16:15:35.430834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423499 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423502 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423505 2564 flags.go:64] FLAG: --seccomp-default="false" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423508 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423510 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423513 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423517 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423520 2564 flags.go:64] FLAG: --storage-driver-password="root" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423523 2564 flags.go:64] FLAG: --storage-driver-secure="false" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423544 2564 flags.go:64] FLAG: --storage-driver-table="stats" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423548 2564 flags.go:64] FLAG: --storage-driver-user="root" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423552 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423555 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423558 2564 flags.go:64] FLAG: --system-cgroups="" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423561 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423567 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423570 2564 flags.go:64] FLAG: --tls-cert-file="" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423573 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423578 2564 flags.go:64] FLAG: --tls-min-version="" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423585 2564 flags.go:64] FLAG: --tls-private-key-file="" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423588 2564 flags.go:64] FLAG: --topology-manager-policy="none" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423590 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423593 2564 flags.go:64] FLAG: --topology-manager-scope="container" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423597 2564 flags.go:64] FLAG: --v="2" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423600 2564 flags.go:64] FLAG: --version="false" Sep 18 16:15:35.431427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423604 2564 flags.go:64] FLAG: --vmodule="" Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423608 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.423611 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423715 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423719 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423722 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423725 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423729 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423732 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423734 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423737 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423739 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423742 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423745 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423747 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423750 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423753 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423755 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423758 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423762 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:15:35.432201 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423766 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423769 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423772 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423775 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423777 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423780 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423784 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423787 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423790 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423793 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423795 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423798 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423800 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423803 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423806 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423809 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423812 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423816 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423819 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:15:35.432803 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423823 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423826 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423828 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423831 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423834 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423836 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423839 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423841 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423844 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423847 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423849 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423860 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423864 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423867 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423870 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423873 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423875 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423878 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423880 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423884 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:15:35.433343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423886 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423889 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423892 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423894 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423897 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423899 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423902 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423904 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423907 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423909 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423912 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423915 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423919 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423921 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423924 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423926 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423929 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423931 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423934 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423937 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:15:35.433871 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423939 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423942 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423944 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423947 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423949 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423952 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423955 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423958 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423960 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423963 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423965 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423969 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423972 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423974 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423976 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423979 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423981 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423984 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423986 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:15:35.434366 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423989 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423991 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423993 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.423996 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.424525 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.431522 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.431537 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.431558 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.431567 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.432995 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433016 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433021 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433025 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433029 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:15:35.434859 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433042 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433048 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433053 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433058 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433063 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433067 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433072 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433076 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433080 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433085 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433089 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433093 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433102 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433106 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433111 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433115 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433119 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433124 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433138 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433143 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:15:35.435195 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433147 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433151 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433155 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433159 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433164 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433189 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433194 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433308 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433321 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433327 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433330 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433333 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433337 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433340 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433345 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433351 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433354 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433358 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433361 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:15:35.435697 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433364 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433367 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433371 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433374 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433377 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433380 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433383 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433386 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433389 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433391 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433394 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433397 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433399 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433402 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433404 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433407 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433410 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433413 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433416 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433419 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:15:35.436174 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433422 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433425 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433428 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433430 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433433 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433436 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433439 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433441 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433444 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433446 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433449 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433451 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433456 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433458 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433461 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433463 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433466 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433469 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433472 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433474 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:15:35.436788 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433477 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433480 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433496 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433500 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433502 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433507 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433511 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433514 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433517 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433520 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433523 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433525 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433528 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433531 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:15:35.437257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433534 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.433539 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433656 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433661 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433664 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433667 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433670 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433673 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433676 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433679 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433681 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433684 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433687 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433689 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433692 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:15:35.437612 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433694 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433698 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433700 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433705 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433707 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433710 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433712 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433716 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433719 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433722 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433725 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433728 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433731 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433733 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433736 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433738 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433741 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433744 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433747 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:15:35.437973 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433749 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433752 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433754 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433757 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433759 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433762 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433764 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433767 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433769 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433772 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433774 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433777 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433779 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433782 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433786 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433789 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433793 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433797 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433800 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433803 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:15:35.438409 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433806 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433808 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433811 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433813 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433816 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433818 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433821 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433823 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433826 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433828 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433831 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433834 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433837 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433839 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433842 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433845 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433847 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433850 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433853 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433855 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:15:35.438915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433858 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433860 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433863 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433866 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433868 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433871 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433875 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433877 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433880 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433883 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433887 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433891 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433895 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433898 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433901 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433903 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433906 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433909 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433911 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433914 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:15:35.439415 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433917 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433920 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433923 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433925 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433928 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433930 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:35.433933 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.433938 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.434062 2564 server.go:962] "Client rotation is on, will bootstrap in background" Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.435965 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.437043 2564 server.go:1019] "Starting client certificate rotation" Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.437141 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.437178 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:15:35.439915 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.439454 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 16:15:35.440526 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.440509 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 16:15:35.448466 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.448450 2564 log.go:25] "Validated CRI v1 runtime API" Sep 18 16:15:35.449925 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.449913 2564 log.go:25] "Validated CRI v1 image API" Sep 18 16:15:35.450791 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.450772 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 18 16:15:35.459329 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.459313 2564 fs.go:135] Filesystem UUIDs: map[6cabd171-220e-42a1-8216-8e394f0912ce:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d75ec6a6-0bff-4817-8a3f-7f9e1a6576dc:/dev/nvme0n1p3] Sep 18 16:15:35.459382 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.459329 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 18 16:15:35.464990 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.464856 2564 manager.go:216] Machine: {Timestamp:2026-09-18 16:15:35.46405054 +0000 UTC m=+1.983318031 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3191681 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22525e4e99371d8e4aed1b2f976af1 SystemUUID:ec22525e-4e99-371d-8e4a-ed1b2f976af1 BootID:d2786a4f-dd3c-4245-bee7-f60efc13f286 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:b1:69:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:b1:69:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:70:ce:b4:7e:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 18 16:15:35.465066 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.464990 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 18 16:15:35.465098 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.465074 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 18 16:15:35.466829 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.466808 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 18 16:15:35.466951 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.466831 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-79.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 18 16:15:35.468138 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.468128 2564 topology_manager.go:138] "Creating topology manager with none policy" Sep 18 16:15:35.468179 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.468141 2564 container_manager_linux.go:309] "Creating device plugin manager" Sep 18 16:15:35.468179 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.468148 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 16:15:35.468179 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.468156 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 16:15:35.468257 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.468215 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 18 16:15:35.471225 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.471214 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 18 16:15:35.471337 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.471328 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 18 16:15:35.471371 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.471363 2564 kubelet.go:486] "Attempting to sync node with API server" Sep 18 16:15:35.471404 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.471382 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 18 16:15:35.471404 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.471392 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 18 16:15:35.471404 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.471400 2564 kubelet.go:398] "Adding apiserver pod source" Sep 18 16:15:35.471477 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.471409 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 18 16:15:35.471837 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.471817 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 16:15:35.471909 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.471853 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 16:15:35.474465 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.474420 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 18 16:15:35.476695 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.476674 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 18 16:15:35.476784 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.476722 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 18 16:15:35.476875 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.476861 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 16:15:35.478133 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478120 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478136 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478142 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478147 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478155 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478160 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478173 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478178 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478184 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478189 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478197 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 18 16:15:35.478201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478202 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 18 16:15:35.478496 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478223 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 18 16:15:35.478496 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.478228 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 18 16:15:35.481989 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.481976 2564 server.go:1268] "Started kubelet" Sep 18 16:15:35.482421 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.482387 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 18 16:15:35.482461 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.482444 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 18 16:15:35.482683 ip-10-0-143-79 systemd[1]: Started Kubernetes Kubelet. Sep 18 16:15:35.483037 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.482764 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 18 16:15:35.483543 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.483409 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:15:35.483543 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.483459 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 18 16:15:35.483543 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.483496 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:15:35.483543 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.483534 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-79.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:15:35.484453 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.484438 2564 server.go:310] "Adding debug handlers to kubelet server" Sep 18 16:15:35.491223 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.491204 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 18 16:15:35.491478 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.491464 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 18 16:15:35.491478 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.491476 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 18 16:15:35.491599 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.491553 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:35.492094 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492057 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 18 16:15:35.492194 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492184 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 18 16:15:35.492330 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492316 2564 factory.go:55] Registering systemd factory Sep 18 16:15:35.492390 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492335 2564 factory.go:223] Registration of the systemd container factory successfully Sep 18 16:15:35.492390 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492057 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 18 16:15:35.492541 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492529 2564 reconstruct.go:97] "Volume reconstruction finished" Sep 18 16:15:35.492541 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492542 2564 reconciler.go:29] "Reconciler: start to sync state" Sep 18 16:15:35.492647 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492588 2564 factory.go:153] Registering CRI-O factory Sep 18 16:15:35.492647 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492602 2564 factory.go:223] Registration of the crio container factory successfully Sep 18 16:15:35.492730 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492652 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 18 16:15:35.492730 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492671 2564 factory.go:103] Registering Raw factory Sep 18 16:15:35.492730 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.492683 2564 manager.go:1196] Started watching for new ooms in manager Sep 18 16:15:35.493188 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.493174 2564 manager.go:349] Starting recovery of all containers Sep 18 16:15:35.494630 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.494593 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 18 16:15:35.494713 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.494638 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:15:35.518292 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.518186 2564 manager.go:354] Recovery completed Sep 18 16:15:35.523252 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.523238 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:15:35.525167 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.525151 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:15:35.525241 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.525179 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:15:35.525241 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.525189 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:15:35.525650 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.525637 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 18 16:15:35.525650 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.525647 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:15:35.525650 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.525650 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 18 16:15:35.525764 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.525667 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 18 16:15:35.526300 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.526287 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:15:35.526341 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.526311 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:15:35.526341 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.526321 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:15:35.526597 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.526582 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fc48f" Sep 18 16:15:35.527206 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.527196 2564 policy_none.go:49] "None policy: Start" Sep 18 16:15:35.527251 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.527209 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 18 16:15:35.527251 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.527220 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 18 16:15:35.528319 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.528309 2564 policy_none.go:47] "Start" Sep 18 16:15:35.532524 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.532511 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fc48f" Sep 18 16:15:35.537046 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.535953 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-79.ec2.internal.18d676725cd1cf54 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-79.ec2.internal,UID:ip-10-0-143-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-79.ec2.internal,},FirstTimestamp:2026-09-18 16:15:35.481954132 +0000 UTC m=+2.001221626,LastTimestamp:2026-09-18 16:15:35.481954132 +0000 UTC m=+2.001221626,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-79.ec2.internal,}" Sep 18 16:15:35.565142 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.565129 2564 manager.go:337] "Starting Device Plugin manager" Sep 18 16:15:35.578261 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.565191 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 18 16:15:35.578261 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.565203 2564 server.go:87] "Starting device plugin registration server" Sep 18 16:15:35.578261 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.565429 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 18 16:15:35.578261 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.565445 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 18 16:15:35.578261 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.565550 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 18 16:15:35.578261 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.565622 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 18 16:15:35.578261 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.565630 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 18 16:15:35.578261 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.566061 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 18 16:15:35.578261 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.566112 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:35.632788 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.632764 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 18 16:15:35.633871 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.633852 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 18 16:15:35.633958 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.633893 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 18 16:15:35.633958 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.633910 2564 kubelet.go:2448] "Starting kubelet main sync loop" Sep 18 16:15:35.633958 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.633945 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 18 16:15:35.636296 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.636275 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:15:35.666103 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.666085 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:15:35.666792 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.666779 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:15:35.666887 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.666808 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:15:35.666887 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.666822 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:15:35.666887 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.666848 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.676915 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.676876 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.676915 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.676895 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-79.ec2.internal\": node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:35.733778 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.733762 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:35.734743 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.734727 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal"] Sep 18 16:15:35.734809 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.734790 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:15:35.736604 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.736592 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:15:35.736680 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.736618 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:15:35.736680 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.736632 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:15:35.738173 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.738159 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:15:35.738278 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.738265 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.738322 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.738288 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:15:35.738837 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.738819 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:15:35.738923 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.738822 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:15:35.738923 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.738859 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:15:35.738923 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.738873 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:15:35.738923 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.738874 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:15:35.738923 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.738887 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:15:35.740269 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.740253 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.740316 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.740285 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:15:35.740945 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.740928 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:15:35.741029 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.740959 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:15:35.741029 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.740972 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:15:35.765031 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.765013 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-79.ec2.internal\" not found" node="ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.769148 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.769133 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-79.ec2.internal\" not found" node="ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.794605 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.794579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb5b1d0ce8905d6c5e18ff0c900c2114-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal\" (UID: \"fb5b1d0ce8905d6c5e18ff0c900c2114\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.794694 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.794614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb5b1d0ce8905d6c5e18ff0c900c2114-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal\" (UID: \"fb5b1d0ce8905d6c5e18ff0c900c2114\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.794694 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.794649 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a6af8d9bebf834f6d415238095457ce-config\") pod \"kube-apiserver-proxy-ip-10-0-143-79.ec2.internal\" (UID: \"6a6af8d9bebf834f6d415238095457ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.834801 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.834784 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:35.895816 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.895796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb5b1d0ce8905d6c5e18ff0c900c2114-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal\" (UID: \"fb5b1d0ce8905d6c5e18ff0c900c2114\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.895888 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.895820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a6af8d9bebf834f6d415238095457ce-config\") pod \"kube-apiserver-proxy-ip-10-0-143-79.ec2.internal\" (UID: \"6a6af8d9bebf834f6d415238095457ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.895888 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.895847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb5b1d0ce8905d6c5e18ff0c900c2114-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal\" (UID: \"fb5b1d0ce8905d6c5e18ff0c900c2114\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.895888 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.895877 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb5b1d0ce8905d6c5e18ff0c900c2114-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal\" (UID: \"fb5b1d0ce8905d6c5e18ff0c900c2114\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.895991 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.895918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb5b1d0ce8905d6c5e18ff0c900c2114-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal\" (UID: \"fb5b1d0ce8905d6c5e18ff0c900c2114\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.895991 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:35.895940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a6af8d9bebf834f6d415238095457ce-config\") pod \"kube-apiserver-proxy-ip-10-0-143-79.ec2.internal\" (UID: \"6a6af8d9bebf834f6d415238095457ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal" Sep 18 16:15:35.935922 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:35.935882 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:36.036688 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:36.036663 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:36.068836 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.068815 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" Sep 18 16:15:36.073253 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.073111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal" Sep 18 16:15:36.137602 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:36.137576 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:36.238132 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:36.238078 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:36.338609 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:36.338587 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:36.437101 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.437076 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 18 16:15:36.437548 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.437198 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 16:15:36.439236 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:36.439214 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:36.537125 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.537086 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:10:35 +0000 UTC" deadline="2028-06-03 20:22:13.354781342 +0000 UTC" Sep 18 16:15:36.537125 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.537120 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14980h6m36.817664391s" Sep 18 16:15:36.540233 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:36.540207 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:36.641014 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:36.640987 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-79.ec2.internal\" not found" Sep 18 16:15:36.684386 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.684362 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:15:36.691746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.691726 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" Sep 18 16:15:36.704266 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:36.704240 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6af8d9bebf834f6d415238095457ce.slice/crio-f2babd22b69f7c4c3972bc11b3cd4198d41ee66615697fdc5d28054325e1a516 WatchSource:0}: Error finding container f2babd22b69f7c4c3972bc11b3cd4198d41ee66615697fdc5d28054325e1a516: Status 404 returned error can't find the container with id f2babd22b69f7c4c3972bc11b3cd4198d41ee66615697fdc5d28054325e1a516 Sep 18 16:15:36.704588 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:36.704570 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5b1d0ce8905d6c5e18ff0c900c2114.slice/crio-0deb4adc03c360567fea6dafd660e7a2c19438550e0c42afc250440f59fe2d87 WatchSource:0}: Error finding container 0deb4adc03c360567fea6dafd660e7a2c19438550e0c42afc250440f59fe2d87: Status 404 returned error can't find the container with id 0deb4adc03c360567fea6dafd660e7a2c19438550e0c42afc250440f59fe2d87 Sep 18 16:15:36.705194 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.705180 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 16:15:36.707124 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.707101 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal" Sep 18 16:15:36.711119 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.711104 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:15:36.715922 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.715905 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 16:15:36.773589 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.773530 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:15:36.960726 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:36.960701 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:15:37.472807 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.472777 2564 apiserver.go:52] "Watching apiserver" Sep 18 16:15:37.480063 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.480038 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 18 16:15:37.480704 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.480623 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-r4pj7","openshift-network-diagnostics/network-check-target-7xjxp","openshift-network-operator/iptables-alerter-wkmhp","openshift-ovn-kubernetes/ovnkube-node-w9n9q","kube-system/global-pull-secret-syncer-msncf","kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9s65v","openshift-dns/node-resolver-8vqrx","openshift-image-registry/node-ca-wztrg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal","openshift-multus/multus-f2vz7","kube-system/konnectivity-agent-xplqn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd","openshift-multus/multus-additional-cni-plugins-cffmd"] Sep 18 16:15:37.483308 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.483275 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.485510 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.485450 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:37.485619 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.485591 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:37.486159 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.486137 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:15:37.486252 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.486168 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jrhnt\"" Sep 18 16:15:37.486252 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.486200 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 18 16:15:37.487742 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.487720 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.490959 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.490037 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:15:37.490959 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.490105 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 18 16:15:37.490959 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.490289 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 18 16:15:37.490959 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.490376 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.490959 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.490457 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-55dsj\"" Sep 18 16:15:37.492219 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.492154 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 18 16:15:37.493155 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.492734 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 18 16:15:37.493155 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.492764 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 18 16:15:37.493155 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.492906 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hwhdt\"" Sep 18 16:15:37.493155 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.492996 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 18 16:15:37.493155 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.493024 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 18 16:15:37.494036 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.493982 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 18 16:15:37.494131 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.494117 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 18 16:15:37.498783 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.498535 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:37.498783 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.498595 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:37.500746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.500726 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.501964 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.501945 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 16:15:37.502917 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.502899 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 18 16:15:37.503148 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503128 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 18 16:15:37.503422 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503405 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.503517 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503442 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tkfs8\"" Sep 18 16:15:37.503791 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-run\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.503874 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-var-lib-kubelet\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.503874 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-run-netns\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.503874 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-node-log\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504015 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503893 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-log-socket\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504015 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-env-overrides\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504015 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:37.504015 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.503973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnkt\" (UniqueName: \"kubernetes.io/projected/415ad465-e1a1-4c60-99dd-7c18748d64e8-kube-api-access-qgnkt\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:37.504202 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-modprobe-d\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.504202 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504035 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-host\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.504202 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzgkx\" (UniqueName: \"kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx\") pod \"network-check-target-7xjxp\" (UID: \"02dcbb62-fa0e-491c-9428-59eb7856d6b2\") " pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:37.504202 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-cni-bin\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504202 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-cni-netd\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504202 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-sysctl-d\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.504474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ce707b-f655-4f08-aed9-315baa4ac890-tmp\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.504474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dn2\" (UniqueName: \"kubernetes.io/projected/81ce707b-f655-4f08-aed9-315baa4ac890-kube-api-access-c5dn2\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.504474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fead61ad-011c-4b2a-81f1-0967e1811606-iptables-alerter-script\") pod \"iptables-alerter-wkmhp\" (UID: \"fead61ad-011c-4b2a-81f1-0967e1811606\") " pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.504474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wns9h\" (UniqueName: \"kubernetes.io/projected/fead61ad-011c-4b2a-81f1-0967e1811606-kube-api-access-wns9h\") pod \"iptables-alerter-wkmhp\" (UID: \"fead61ad-011c-4b2a-81f1-0967e1811606\") " pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.504474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-kubernetes\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.504474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504374 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-etc-openvswitch\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-run-openvswitch\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-sysctl-conf\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.504474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-systemd\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.504893 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-systemd-units\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504893 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-ovnkube-config\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504893 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-sys\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.504893 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fead61ad-011c-4b2a-81f1-0967e1811606-host-slash\") pod \"iptables-alerter-wkmhp\" (UID: \"fead61ad-011c-4b2a-81f1-0967e1811606\") " pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.504893 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-run-ovn\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504893 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-run-ovn-kubernetes\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504893 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-ovnkube-script-lib\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504893 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-run-systemd\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504893 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-var-lib-openvswitch\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.504893 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-sysconfig\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.505339 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81ce707b-f655-4f08-aed9-315baa4ac890-etc-tuned\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.505339 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504933 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-kubelet\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.505339 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.505339 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.504997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-ovn-node-metrics-cert\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.505339 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.505041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrxtb\" (UniqueName: \"kubernetes.io/projected/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-kube-api-access-mrxtb\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.505339 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.505064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-lib-modules\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.505339 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.505084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-slash\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.505681 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.505643 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 18 16:15:37.505740 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.505693 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-trwtz\"" Sep 18 16:15:37.506026 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.506006 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 18 16:15:37.506134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.506117 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 18 16:15:37.506363 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.506345 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.508768 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.508550 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 18 16:15:37.508768 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.508680 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:37.508863 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.508762 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:37.509186 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.509165 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wl54c\"" Sep 18 16:15:37.509387 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.509277 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 18 16:15:37.509468 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.509388 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 18 16:15:37.511670 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.511276 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.513251 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.513233 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 18 16:15:37.513626 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.513604 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 18 16:15:37.513731 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.513705 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 18 16:15:37.513731 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.513719 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 18 16:15:37.514190 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.514168 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:37.516582 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.516077 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vj79p\"" Sep 18 16:15:37.518149 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.516959 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 18 16:15:37.518149 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.517045 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 18 16:15:37.518149 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.517297 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6kklg\"" Sep 18 16:15:37.518580 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.518259 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.520521 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.520365 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 18 16:15:37.520521 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.520389 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 18 16:15:37.520716 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.520646 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fllbm\"" Sep 18 16:15:37.543987 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.543966 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fswng" Sep 18 16:15:37.551522 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.551502 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fswng" Sep 18 16:15:37.593602 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.593582 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 18 16:15:37.606165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-slash\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.606165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.606323 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606181 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-multus-socket-dir-parent\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.606323 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-run-k8s-cni-cncf-io\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.606323 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-run-netns\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.606323 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4-konnectivity-ca\") pod \"konnectivity-agent-xplqn\" (UID: \"17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4\") " pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:37.606323 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-slash\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.606323 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-var-lib-kubelet\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.606626 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-node-log\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.606626 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606335 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-var-lib-kubelet\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.606626 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-env-overrides\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.606626 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:37.606626 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnkt\" (UniqueName: \"kubernetes.io/projected/415ad465-e1a1-4c60-99dd-7c18748d64e8-kube-api-access-qgnkt\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:37.606626 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-cnibin\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.606626 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.606533 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:37.606626 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.606610 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs podName:415ad465-e1a1-4c60-99dd-7c18748d64e8 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:38.106583835 +0000 UTC m=+4.625851331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs") pod "network-metrics-daemon-r4pj7" (UID: "415ad465-e1a1-4c60-99dd-7c18748d64e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-modprobe-d\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkx\" (UniqueName: \"kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx\") pod \"network-check-target-7xjxp\" (UID: \"02dcbb62-fa0e-491c-9428-59eb7856d6b2\") " pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606697 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-node-log\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-cni-netd\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-modprobe-d\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606768 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-cni-netd\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-sysctl-d\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ce707b-f655-4f08-aed9-315baa4ac890-tmp\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606840 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dn2\" (UniqueName: \"kubernetes.io/projected/81ce707b-f655-4f08-aed9-315baa4ac890-kube-api-access-c5dn2\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wns9h\" (UniqueName: \"kubernetes.io/projected/fead61ad-011c-4b2a-81f1-0967e1811606-kube-api-access-wns9h\") pod \"iptables-alerter-wkmhp\" (UID: \"fead61ad-011c-4b2a-81f1-0967e1811606\") " pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxj6x\" (UniqueName: \"kubernetes.io/projected/931320fc-4a72-46a3-94f2-431ff1c66526-kube-api-access-rxj6x\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4-agent-certs\") pod \"konnectivity-agent-xplqn\" (UID: \"17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4\") " pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-sysctl-d\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e1fc1cd-109e-4bb8-aa37-19c71590b563-serviceca\") pod \"node-ca-wztrg\" (UID: \"5e1fc1cd-109e-4bb8-aa37-19c71590b563\") " pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.606982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.606937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-env-overrides\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e1fc1cd-109e-4bb8-aa37-19c71590b563-host\") pod \"node-ca-wztrg\" (UID: \"5e1fc1cd-109e-4bb8-aa37-19c71590b563\") " pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/992c20b0-878a-4325-82d7-10fa9f9acd06-tmp-dir\") pod \"node-resolver-8vqrx\" (UID: \"992c20b0-878a-4325-82d7-10fa9f9acd06\") " pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-socket-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607129 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-system-cni-dir\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-cnibin\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fead61ad-011c-4b2a-81f1-0967e1811606-host-slash\") pod \"iptables-alerter-wkmhp\" (UID: \"fead61ad-011c-4b2a-81f1-0967e1811606\") " pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607265 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-run-ovn\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-run-ovn-kubernetes\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fead61ad-011c-4b2a-81f1-0967e1811606-host-slash\") pod \"iptables-alerter-wkmhp\" (UID: \"fead61ad-011c-4b2a-81f1-0967e1811606\") " pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w9s4\" (UniqueName: \"kubernetes.io/projected/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-kube-api-access-6w9s4\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-run-systemd\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607373 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-run-ovn\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-sysconfig\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81ce707b-f655-4f08-aed9-315baa4ac890-etc-tuned\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-run-ovn-kubernetes\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.607746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607430 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-run-systemd\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-multus-cni-dir\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607474 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-lib-modules\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-sysconfig\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-var-lib-cni-multus\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607607 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-lib-modules\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-run\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-run-netns\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607671 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-run\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-log-socket\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-run-netns\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607763 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-log-socket\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/458d51a9-e012-413e-8dd6-8354a56b8abc-dbus\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607821 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-registration-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/931320fc-4a72-46a3-94f2-431ff1c66526-cni-binary-copy\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.608574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607863 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-etc-kubernetes\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-host\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-cni-bin\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/458d51a9-e012-413e-8dd6-8354a56b8abc-kubelet-config\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-cni-bin\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.607980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fead61ad-011c-4b2a-81f1-0967e1811606-iptables-alerter-script\") pod \"iptables-alerter-wkmhp\" (UID: \"fead61ad-011c-4b2a-81f1-0967e1811606\") " pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjnff\" (UniqueName: \"kubernetes.io/projected/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-kube-api-access-bjnff\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-kubernetes\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-etc-openvswitch\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-run-openvswitch\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5xtt\" (UniqueName: \"kubernetes.io/projected/5e1fc1cd-109e-4bb8-aa37-19c71590b563-kube-api-access-k5xtt\") pod \"node-ca-wztrg\" (UID: \"5e1fc1cd-109e-4bb8-aa37-19c71590b563\") " pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/992c20b0-878a-4325-82d7-10fa9f9acd06-hosts-file\") pod \"node-resolver-8vqrx\" (UID: \"992c20b0-878a-4325-82d7-10fa9f9acd06\") " pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bs7b\" (UniqueName: \"kubernetes.io/projected/992c20b0-878a-4325-82d7-10fa9f9acd06-kube-api-access-8bs7b\") pod \"node-resolver-8vqrx\" (UID: \"992c20b0-878a-4325-82d7-10fa9f9acd06\") " pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-kubernetes\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608019 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-host\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608258 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-etc-openvswitch\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.609333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608280 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-run-openvswitch\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-sys-fs\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-var-lib-kubelet\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-sysctl-conf\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-systemd\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-systemd-units\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-ovnkube-config\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-var-lib-cni-bin\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608496 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-sysctl-conf\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-multus-conf-dir\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608543 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-systemd-units\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/931320fc-4a72-46a3-94f2-431ff1c66526-multus-daemon-config\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-sys\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-ovnkube-script-lib\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-etc-systemd\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-hostroot\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fead61ad-011c-4b2a-81f1-0967e1811606-iptables-alerter-script\") pod \"iptables-alerter-wkmhp\" (UID: \"fead61ad-011c-4b2a-81f1-0967e1811606\") " pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.610165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-run-multus-certs\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ce707b-f655-4f08-aed9-315baa4ac890-sys\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-system-cni-dir\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-var-lib-openvswitch\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-kubelet\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608786 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-ovn-node-metrics-cert\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrxtb\" (UniqueName: \"kubernetes.io/projected/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-kube-api-access-mrxtb\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.608869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-device-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.609041 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-var-lib-openvswitch\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.609080 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-kubelet\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.609123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-ovnkube-config\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.609152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-ovnkube-script-lib\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.609126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.610954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.610542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81ce707b-f655-4f08-aed9-315baa4ac890-etc-tuned\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.611694 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.611662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-ovn-node-metrics-cert\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.612422 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.612402 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81ce707b-f655-4f08-aed9-315baa4ac890-tmp\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.616082 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.616059 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:15:37.616082 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.616084 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:15:37.616232 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.616097 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzgkx for pod openshift-network-diagnostics/network-check-target-7xjxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:37.616232 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.616161 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx podName:02dcbb62-fa0e-491c-9428-59eb7856d6b2 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:38.116142993 +0000 UTC m=+4.635410668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgkx" (UniqueName: "kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx") pod "network-check-target-7xjxp" (UID: "02dcbb62-fa0e-491c-9428-59eb7856d6b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:37.617043 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.616987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wns9h\" (UniqueName: \"kubernetes.io/projected/fead61ad-011c-4b2a-81f1-0967e1811606-kube-api-access-wns9h\") pod \"iptables-alerter-wkmhp\" (UID: \"fead61ad-011c-4b2a-81f1-0967e1811606\") " pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.619204 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.619178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dn2\" (UniqueName: \"kubernetes.io/projected/81ce707b-f655-4f08-aed9-315baa4ac890-kube-api-access-c5dn2\") pod \"tuned-9s65v\" (UID: \"81ce707b-f655-4f08-aed9-315baa4ac890\") " pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.619720 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.619700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrxtb\" (UniqueName: \"kubernetes.io/projected/45cc0900-9bb9-4d4e-9ca6-7f19aa39807f-kube-api-access-mrxtb\") pod \"ovnkube-node-w9n9q\" (UID: \"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f\") " pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.621109 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.621088 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnkt\" (UniqueName: \"kubernetes.io/projected/415ad465-e1a1-4c60-99dd-7c18748d64e8-kube-api-access-qgnkt\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:37.639536 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.639476 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" event={"ID":"fb5b1d0ce8905d6c5e18ff0c900c2114","Type":"ContainerStarted","Data":"0deb4adc03c360567fea6dafd660e7a2c19438550e0c42afc250440f59fe2d87"} Sep 18 16:15:37.640566 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.640545 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal" event={"ID":"6a6af8d9bebf834f6d415238095457ce","Type":"ContainerStarted","Data":"f2babd22b69f7c4c3972bc11b3cd4198d41ee66615697fdc5d28054325e1a516"} Sep 18 16:15:37.709404 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/458d51a9-e012-413e-8dd6-8354a56b8abc-kubelet-config\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:37.709533 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnff\" (UniqueName: \"kubernetes.io/projected/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-kube-api-access-bjnff\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.709533 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5xtt\" (UniqueName: \"kubernetes.io/projected/5e1fc1cd-109e-4bb8-aa37-19c71590b563-kube-api-access-k5xtt\") pod \"node-ca-wztrg\" (UID: \"5e1fc1cd-109e-4bb8-aa37-19c71590b563\") " pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.709533 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/992c20b0-878a-4325-82d7-10fa9f9acd06-hosts-file\") pod \"node-resolver-8vqrx\" (UID: \"992c20b0-878a-4325-82d7-10fa9f9acd06\") " pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.709533 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bs7b\" (UniqueName: \"kubernetes.io/projected/992c20b0-878a-4325-82d7-10fa9f9acd06-kube-api-access-8bs7b\") pod \"node-resolver-8vqrx\" (UID: \"992c20b0-878a-4325-82d7-10fa9f9acd06\") " pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.709533 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-sys-fs\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.709533 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-var-lib-kubelet\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/458d51a9-e012-413e-8dd6-8354a56b8abc-kubelet-config\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709550 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/992c20b0-878a-4325-82d7-10fa9f9acd06-hosts-file\") pod \"node-resolver-8vqrx\" (UID: \"992c20b0-878a-4325-82d7-10fa9f9acd06\") " pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-var-lib-kubelet\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-sys-fs\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-var-lib-cni-bin\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-multus-conf-dir\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/931320fc-4a72-46a3-94f2-431ff1c66526-multus-daemon-config\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-hostroot\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-multus-conf-dir\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-run-multus-certs\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709737 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-var-lib-cni-bin\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709739 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-run-multus-certs\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-hostroot\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-system-cni-dir\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-system-cni-dir\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-device-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.709826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-multus-socket-dir-parent\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.709992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-run-k8s-cni-cncf-io\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-run-netns\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710063 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-device-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4-konnectivity-ca\") pod \"konnectivity-agent-xplqn\" (UID: \"17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4\") " pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710099 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-run-k8s-cni-cncf-io\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-cnibin\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710147 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-multus-socket-dir-parent\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxj6x\" (UniqueName: \"kubernetes.io/projected/931320fc-4a72-46a3-94f2-431ff1c66526-kube-api-access-rxj6x\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/931320fc-4a72-46a3-94f2-431ff1c66526-multus-daemon-config\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4-agent-certs\") pod \"konnectivity-agent-xplqn\" (UID: \"17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4\") " pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e1fc1cd-109e-4bb8-aa37-19c71590b563-serviceca\") pod \"node-ca-wztrg\" (UID: \"5e1fc1cd-109e-4bb8-aa37-19c71590b563\") " pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e1fc1cd-109e-4bb8-aa37-19c71590b563-host\") pod \"node-ca-wztrg\" (UID: \"5e1fc1cd-109e-4bb8-aa37-19c71590b563\") " pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710281 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-run-netns\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.710658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/992c20b0-878a-4325-82d7-10fa9f9acd06-tmp-dir\") pod \"node-resolver-8vqrx\" (UID: \"992c20b0-878a-4325-82d7-10fa9f9acd06\") " pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-socket-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-system-cni-dir\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-cnibin\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w9s4\" (UniqueName: \"kubernetes.io/projected/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-kube-api-access-6w9s4\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-multus-cni-dir\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710558 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710577 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-multus-cni-dir\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710593 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/992c20b0-878a-4325-82d7-10fa9f9acd06-tmp-dir\") pod \"node-resolver-8vqrx\" (UID: \"992c20b0-878a-4325-82d7-10fa9f9acd06\") " pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710625 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4-konnectivity-ca\") pod \"konnectivity-agent-xplqn\" (UID: \"17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4\") " pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-var-lib-cni-multus\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710671 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-socket-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710674 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-cnibin\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.710679 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-cnibin\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.711501 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/458d51a9-e012-413e-8dd6-8354a56b8abc-dbus\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-registration-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:37.710741 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret podName:458d51a9-e012-413e-8dd6-8354a56b8abc nodeName:}" failed. No retries permitted until 2026-09-18 16:15:38.210723541 +0000 UTC m=+4.729991022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret") pod "global-pull-secret-syncer-msncf" (UID: "458d51a9-e012-413e-8dd6-8354a56b8abc") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-registration-dir\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/931320fc-4a72-46a3-94f2-431ff1c66526-cni-binary-copy\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710791 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-system-cni-dir\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-etc-kubernetes\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/458d51a9-e012-413e-8dd6-8354a56b8abc-dbus\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e1fc1cd-109e-4bb8-aa37-19c71590b563-host\") pod \"node-ca-wztrg\" (UID: \"5e1fc1cd-109e-4bb8-aa37-19c71590b563\") " pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-host-var-lib-cni-multus\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/931320fc-4a72-46a3-94f2-431ff1c66526-etc-kubernetes\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.710927 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.711073 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.711313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e1fc1cd-109e-4bb8-aa37-19c71590b563-serviceca\") pod \"node-ca-wztrg\" (UID: \"5e1fc1cd-109e-4bb8-aa37-19c71590b563\") " pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.711338 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/931320fc-4a72-46a3-94f2-431ff1c66526-cni-binary-copy\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.712130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.711362 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.713195 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.713176 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4-agent-certs\") pod \"konnectivity-agent-xplqn\" (UID: \"17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4\") " pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:37.738180 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.738107 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w9s4\" (UniqueName: \"kubernetes.io/projected/06f6e7c9-8a1c-40fd-a2ba-643769658c5e-kube-api-access-6w9s4\") pod \"aws-ebs-csi-driver-node-zj5bd\" (UID: \"06f6e7c9-8a1c-40fd-a2ba-643769658c5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.738270 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.738238 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bs7b\" (UniqueName: \"kubernetes.io/projected/992c20b0-878a-4325-82d7-10fa9f9acd06-kube-api-access-8bs7b\") pod \"node-resolver-8vqrx\" (UID: \"992c20b0-878a-4325-82d7-10fa9f9acd06\") " pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.738954 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.738915 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5xtt\" (UniqueName: \"kubernetes.io/projected/5e1fc1cd-109e-4bb8-aa37-19c71590b563-kube-api-access-k5xtt\") pod \"node-ca-wztrg\" (UID: \"5e1fc1cd-109e-4bb8-aa37-19c71590b563\") " pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.739086 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.739067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnff\" (UniqueName: \"kubernetes.io/projected/4ae800b0-36f5-4c25-b07c-cd35de96f0d8-kube-api-access-bjnff\") pod \"multus-additional-cni-plugins-cffmd\" (UID: \"4ae800b0-36f5-4c25-b07c-cd35de96f0d8\") " pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:37.739153 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.739126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxj6x\" (UniqueName: \"kubernetes.io/projected/931320fc-4a72-46a3-94f2-431ff1c66526-kube-api-access-rxj6x\") pod \"multus-f2vz7\" (UID: \"931320fc-4a72-46a3-94f2-431ff1c66526\") " pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.804737 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.804713 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9s65v" Sep 18 16:15:37.812552 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.812535 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wkmhp" Sep 18 16:15:37.822148 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.822130 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:15:37.828760 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.828742 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8vqrx" Sep 18 16:15:37.836295 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.836278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wztrg" Sep 18 16:15:37.842751 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.842736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" Sep 18 16:15:37.850361 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.850342 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f2vz7" Sep 18 16:15:37.857876 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.857850 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:37.863417 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:37.863395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cffmd" Sep 18 16:15:38.036134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.036062 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:15:38.113313 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.113282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:38.113469 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:38.113427 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:38.113562 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:38.113512 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs podName:415ad465-e1a1-4c60-99dd-7c18748d64e8 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:39.113480121 +0000 UTC m=+5.632747645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs") pod "network-metrics-daemon-r4pj7" (UID: "415ad465-e1a1-4c60-99dd-7c18748d64e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:38.214525 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.214463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:38.214669 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.214557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkx\" (UniqueName: \"kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx\") pod \"network-check-target-7xjxp\" (UID: \"02dcbb62-fa0e-491c-9428-59eb7856d6b2\") " pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:38.214732 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:38.214689 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:38.214732 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:38.214702 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:15:38.214732 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:38.214719 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:15:38.214732 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:38.214733 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzgkx for pod openshift-network-diagnostics/network-check-target-7xjxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:38.214915 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:38.214760 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret podName:458d51a9-e012-413e-8dd6-8354a56b8abc nodeName:}" failed. No retries permitted until 2026-09-18 16:15:39.214741371 +0000 UTC m=+5.734008871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret") pod "global-pull-secret-syncer-msncf" (UID: "458d51a9-e012-413e-8dd6-8354a56b8abc") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:38.214915 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:38.214779 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx podName:02dcbb62-fa0e-491c-9428-59eb7856d6b2 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:39.214767877 +0000 UTC m=+5.734035370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgkx" (UniqueName: "kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx") pod "network-check-target-7xjxp" (UID: "02dcbb62-fa0e-491c-9428-59eb7856d6b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:38.554127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.554091 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:10:37 +0000 UTC" deadline="2028-02-12 13:27:37.501272053 +0000 UTC" Sep 18 16:15:38.554127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.554126 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12285h11m58.947149893s" Sep 18 16:15:38.555950 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:38.555928 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cc0900_9bb9_4d4e_9ca6_7f19aa39807f.slice/crio-be12b3823781933ba1acacdbf042071584b23a37bf37dfc7c3e65872b7cc8d7e WatchSource:0}: Error finding container be12b3823781933ba1acacdbf042071584b23a37bf37dfc7c3e65872b7cc8d7e: Status 404 returned error can't find the container with id be12b3823781933ba1acacdbf042071584b23a37bf37dfc7c3e65872b7cc8d7e Sep 18 16:15:38.557760 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:38.557689 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ce707b_f655_4f08_aed9_315baa4ac890.slice/crio-1972a70f7c008146573045c27daa9a8aee93184ec88d179a89eb61fdb7469b0c WatchSource:0}: Error finding container 1972a70f7c008146573045c27daa9a8aee93184ec88d179a89eb61fdb7469b0c: Status 404 returned error can't find the container with id 1972a70f7c008146573045c27daa9a8aee93184ec88d179a89eb61fdb7469b0c Sep 18 16:15:38.559460 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:38.559439 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae800b0_36f5_4c25_b07c_cd35de96f0d8.slice/crio-f595559cbc8e8fd5fa28edc1c4dabe598a85ab4d035224662e31ee070c4de289 WatchSource:0}: Error finding container f595559cbc8e8fd5fa28edc1c4dabe598a85ab4d035224662e31ee070c4de289: Status 404 returned error can't find the container with id f595559cbc8e8fd5fa28edc1c4dabe598a85ab4d035224662e31ee070c4de289 Sep 18 16:15:38.563849 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:38.563370 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931320fc_4a72_46a3_94f2_431ff1c66526.slice/crio-278e5c6ceca6bb016aede698fff0adf99c98be7dffda6434a62f001b56042681 WatchSource:0}: Error finding container 278e5c6ceca6bb016aede698fff0adf99c98be7dffda6434a62f001b56042681: Status 404 returned error can't find the container with id 278e5c6ceca6bb016aede698fff0adf99c98be7dffda6434a62f001b56042681 Sep 18 16:15:38.566713 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:38.566692 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1fc1cd_109e_4bb8_aa37_19c71590b563.slice/crio-ca9b5485a719e7ec1bb2498f123691cefda87b23c0ef6614d73841003e7fe20c WatchSource:0}: Error finding container ca9b5485a719e7ec1bb2498f123691cefda87b23c0ef6614d73841003e7fe20c: Status 404 returned error can't find the container with id ca9b5485a719e7ec1bb2498f123691cefda87b23c0ef6614d73841003e7fe20c Sep 18 16:15:38.567599 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:38.567576 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfead61ad_011c_4b2a_81f1_0967e1811606.slice/crio-6dfc747e2e6c1cccd8a729d98d4b069c5743df2776f5f52ab66b8d66ac92ed53 WatchSource:0}: Error finding container 6dfc747e2e6c1cccd8a729d98d4b069c5743df2776f5f52ab66b8d66ac92ed53: Status 404 returned error can't find the container with id 6dfc747e2e6c1cccd8a729d98d4b069c5743df2776f5f52ab66b8d66ac92ed53 Sep 18 16:15:38.569373 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:15:38.568688 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f6e7c9_8a1c_40fd_a2ba_643769658c5e.slice/crio-19d8c5cb9e123141625979701733281094b543060b7aef1d4a99108c3ca97596 WatchSource:0}: Error finding container 19d8c5cb9e123141625979701733281094b543060b7aef1d4a99108c3ca97596: Status 404 returned error can't find the container with id 19d8c5cb9e123141625979701733281094b543060b7aef1d4a99108c3ca97596 Sep 18 16:15:38.634359 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.634247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:38.634448 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:38.634431 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:38.646945 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.646922 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal" event={"ID":"6a6af8d9bebf834f6d415238095457ce","Type":"ContainerStarted","Data":"33891894d39a895fbe5fa00b3be8476b72d6d50cb243ef5c7a06671b95f5a9f6"} Sep 18 16:15:38.650538 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.650518 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8vqrx" event={"ID":"992c20b0-878a-4325-82d7-10fa9f9acd06","Type":"ContainerStarted","Data":"c012fd8fde11cb9975f1c28b2af1dbea520d75f725e4a3d0279f64ba4b3b4884"} Sep 18 16:15:38.653691 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.653668 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wztrg" event={"ID":"5e1fc1cd-109e-4bb8-aa37-19c71590b563","Type":"ContainerStarted","Data":"ca9b5485a719e7ec1bb2498f123691cefda87b23c0ef6614d73841003e7fe20c"} Sep 18 16:15:38.655317 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.655294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9s65v" event={"ID":"81ce707b-f655-4f08-aed9-315baa4ac890","Type":"ContainerStarted","Data":"1972a70f7c008146573045c27daa9a8aee93184ec88d179a89eb61fdb7469b0c"} Sep 18 16:15:38.656601 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.656581 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" event={"ID":"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f","Type":"ContainerStarted","Data":"be12b3823781933ba1acacdbf042071584b23a37bf37dfc7c3e65872b7cc8d7e"} Sep 18 16:15:38.659721 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.659702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xplqn" event={"ID":"17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4","Type":"ContainerStarted","Data":"3d0a74c2a6ace1dd8e0cc9b970dc056262d7ce105b02290abdfbfa4ff9569270"} Sep 18 16:15:38.661185 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.661140 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-79.ec2.internal" podStartSLOduration=2.661125466 podStartE2EDuration="2.661125466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:38.660644486 +0000 UTC m=+5.179911984" watchObservedRunningTime="2026-09-18 16:15:38.661125466 +0000 UTC m=+5.180392965" Sep 18 16:15:38.661442 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.661415 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" event={"ID":"06f6e7c9-8a1c-40fd-a2ba-643769658c5e","Type":"ContainerStarted","Data":"19d8c5cb9e123141625979701733281094b543060b7aef1d4a99108c3ca97596"} Sep 18 16:15:38.664057 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.664028 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wkmhp" event={"ID":"fead61ad-011c-4b2a-81f1-0967e1811606","Type":"ContainerStarted","Data":"6dfc747e2e6c1cccd8a729d98d4b069c5743df2776f5f52ab66b8d66ac92ed53"} Sep 18 16:15:38.668038 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.668010 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f2vz7" event={"ID":"931320fc-4a72-46a3-94f2-431ff1c66526","Type":"ContainerStarted","Data":"278e5c6ceca6bb016aede698fff0adf99c98be7dffda6434a62f001b56042681"} Sep 18 16:15:38.669129 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:38.669108 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cffmd" event={"ID":"4ae800b0-36f5-4c25-b07c-cd35de96f0d8","Type":"ContainerStarted","Data":"f595559cbc8e8fd5fa28edc1c4dabe598a85ab4d035224662e31ee070c4de289"} Sep 18 16:15:39.121706 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:39.121632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:39.121850 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:39.121785 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:39.121903 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:39.121854 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs podName:415ad465-e1a1-4c60-99dd-7c18748d64e8 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:41.121832995 +0000 UTC m=+7.641100490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs") pod "network-metrics-daemon-r4pj7" (UID: "415ad465-e1a1-4c60-99dd-7c18748d64e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:39.222748 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:39.222714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkx\" (UniqueName: \"kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx\") pod \"network-check-target-7xjxp\" (UID: \"02dcbb62-fa0e-491c-9428-59eb7856d6b2\") " pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:39.222906 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:39.222770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:39.222906 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:39.222882 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:39.223012 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:39.222939 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret podName:458d51a9-e012-413e-8dd6-8354a56b8abc nodeName:}" failed. No retries permitted until 2026-09-18 16:15:41.222919726 +0000 UTC m=+7.742187219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret") pod "global-pull-secret-syncer-msncf" (UID: "458d51a9-e012-413e-8dd6-8354a56b8abc") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:39.223079 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:39.223020 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:15:39.223079 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:39.223035 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:15:39.223079 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:39.223049 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzgkx for pod openshift-network-diagnostics/network-check-target-7xjxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:39.223218 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:39.223083 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx podName:02dcbb62-fa0e-491c-9428-59eb7856d6b2 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:41.223072146 +0000 UTC m=+7.742339636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgkx" (UniqueName: "kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx") pod "network-check-target-7xjxp" (UID: "02dcbb62-fa0e-491c-9428-59eb7856d6b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:39.554631 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:39.554495 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:10:37 +0000 UTC" deadline="2028-04-25 10:31:56.579055948 +0000 UTC" Sep 18 16:15:39.554631 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:39.554566 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14034h16m17.024494915s" Sep 18 16:15:39.638141 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:39.638111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:39.638295 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:39.638237 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:39.638658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:39.638638 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:39.638780 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:39.638735 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:39.680508 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:39.680219 2564 generic.go:356] "Generic (PLEG): container finished" podID="fb5b1d0ce8905d6c5e18ff0c900c2114" containerID="944276a2b2cd97f1f8ac75dfda38b00af93f5b817e9f1b400156dd97df44e2d8" exitCode=0 Sep 18 16:15:39.680508 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:39.680314 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" event={"ID":"fb5b1d0ce8905d6c5e18ff0c900c2114","Type":"ContainerDied","Data":"944276a2b2cd97f1f8ac75dfda38b00af93f5b817e9f1b400156dd97df44e2d8"} Sep 18 16:15:40.635771 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:40.635024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:40.635771 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:40.635161 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:40.699695 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:40.699658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" event={"ID":"fb5b1d0ce8905d6c5e18ff0c900c2114","Type":"ContainerStarted","Data":"7989f122de72338539a5a962df2bb714866fa8ff147e0e65c47688a8c4c38ac2"} Sep 18 16:15:41.139553 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:41.139516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:41.139733 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:41.139679 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:41.139790 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:41.139743 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs podName:415ad465-e1a1-4c60-99dd-7c18748d64e8 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:45.139723482 +0000 UTC m=+11.658990972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs") pod "network-metrics-daemon-r4pj7" (UID: "415ad465-e1a1-4c60-99dd-7c18748d64e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:41.240547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:41.240495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkx\" (UniqueName: \"kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx\") pod \"network-check-target-7xjxp\" (UID: \"02dcbb62-fa0e-491c-9428-59eb7856d6b2\") " pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:41.240717 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:41.240557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:41.240717 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:41.240702 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:41.240822 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:41.240757 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret podName:458d51a9-e012-413e-8dd6-8354a56b8abc nodeName:}" failed. No retries permitted until 2026-09-18 16:15:45.240739984 +0000 UTC m=+11.760007464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret") pod "global-pull-secret-syncer-msncf" (UID: "458d51a9-e012-413e-8dd6-8354a56b8abc") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:41.241268 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:41.241151 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:15:41.241268 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:41.241178 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:15:41.241268 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:41.241191 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzgkx for pod openshift-network-diagnostics/network-check-target-7xjxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:41.241268 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:41.241234 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx podName:02dcbb62-fa0e-491c-9428-59eb7856d6b2 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:45.24122046 +0000 UTC m=+11.760487952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgkx" (UniqueName: "kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx") pod "network-check-target-7xjxp" (UID: "02dcbb62-fa0e-491c-9428-59eb7856d6b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:41.635639 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:41.635569 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:41.635797 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:41.635680 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:41.636143 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:41.636116 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:41.636269 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:41.636221 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:42.635138 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:42.635106 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:42.635309 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:42.635231 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:43.634769 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:43.634737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:43.635215 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:43.634904 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:43.635413 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:43.635394 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:43.635566 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:43.635508 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:44.634613 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:44.634571 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:44.634783 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:44.634728 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:45.166110 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:45.166074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:45.166299 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:45.166228 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:45.166299 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:45.166289 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs podName:415ad465-e1a1-4c60-99dd-7c18748d64e8 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:53.166269905 +0000 UTC m=+19.685537387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs") pod "network-metrics-daemon-r4pj7" (UID: "415ad465-e1a1-4c60-99dd-7c18748d64e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:45.266461 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:45.266417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkx\" (UniqueName: \"kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx\") pod \"network-check-target-7xjxp\" (UID: \"02dcbb62-fa0e-491c-9428-59eb7856d6b2\") " pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:45.266627 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:45.266497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:45.266733 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:45.266709 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:15:45.266733 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:45.266722 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:45.266846 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:45.266737 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:15:45.266846 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:45.266750 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzgkx for pod openshift-network-diagnostics/network-check-target-7xjxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:45.266846 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:45.266785 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret podName:458d51a9-e012-413e-8dd6-8354a56b8abc nodeName:}" failed. No retries permitted until 2026-09-18 16:15:53.266766498 +0000 UTC m=+19.786033978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret") pod "global-pull-secret-syncer-msncf" (UID: "458d51a9-e012-413e-8dd6-8354a56b8abc") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:45.266846 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:45.266801 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx podName:02dcbb62-fa0e-491c-9428-59eb7856d6b2 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:53.266792578 +0000 UTC m=+19.786060057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgkx" (UniqueName: "kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx") pod "network-check-target-7xjxp" (UID: "02dcbb62-fa0e-491c-9428-59eb7856d6b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:45.636176 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:45.636106 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:45.636583 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:45.636114 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:45.636583 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:45.636217 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:45.636583 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:45.636301 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:46.635367 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:46.635189 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:46.635367 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:46.635333 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:47.634652 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:47.634625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:47.634652 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:47.634637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:47.635074 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:47.634782 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:47.635074 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:47.634917 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:48.634630 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:48.634530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:48.634802 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:48.634677 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:49.634378 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:49.634342 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:49.634564 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:49.634451 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:49.634564 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:49.634506 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:49.634647 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:49.634597 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:50.634191 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:50.634155 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:50.634632 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:50.634283 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:51.634182 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:51.634147 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:51.634355 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:51.634262 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:51.634355 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:51.634313 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:51.634778 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:51.634411 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:52.634306 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:52.634275 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:52.634535 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:52.634397 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:53.223371 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:53.223336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:53.223581 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:53.223524 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:53.223644 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:53.223597 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs podName:415ad465-e1a1-4c60-99dd-7c18748d64e8 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.223575048 +0000 UTC m=+35.742842538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs") pod "network-metrics-daemon-r4pj7" (UID: "415ad465-e1a1-4c60-99dd-7c18748d64e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:15:53.324204 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:53.324169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkx\" (UniqueName: \"kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx\") pod \"network-check-target-7xjxp\" (UID: \"02dcbb62-fa0e-491c-9428-59eb7856d6b2\") " pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:53.324204 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:53.324208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:53.324405 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:53.324290 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:53.324405 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:53.324309 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:15:53.324405 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:53.324332 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:15:53.324405 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:53.324337 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret podName:458d51a9-e012-413e-8dd6-8354a56b8abc nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.324323543 +0000 UTC m=+35.843591021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret") pod "global-pull-secret-syncer-msncf" (UID: "458d51a9-e012-413e-8dd6-8354a56b8abc") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:15:53.324405 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:53.324343 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzgkx for pod openshift-network-diagnostics/network-check-target-7xjxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:53.324405 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:53.324383 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx podName:02dcbb62-fa0e-491c-9428-59eb7856d6b2 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.3243706 +0000 UTC m=+35.843638098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgkx" (UniqueName: "kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx") pod "network-check-target-7xjxp" (UID: "02dcbb62-fa0e-491c-9428-59eb7856d6b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:15:53.634950 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:53.634876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:53.635395 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:53.635002 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:53.635395 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:53.635047 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:53.635395 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:53.635153 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:54.634133 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:54.634103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:54.634314 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:54.634218 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:55.635900 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.635871 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:55.636325 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:55.635975 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:55.636325 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.636284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:55.636427 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:55.636390 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:55.730884 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.730829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8vqrx" event={"ID":"992c20b0-878a-4325-82d7-10fa9f9acd06","Type":"ContainerStarted","Data":"9cae24e828af788771bc580c4aeea046f67a35967ee585dffe7185172d216857"} Sep 18 16:15:55.733054 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.733000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wztrg" event={"ID":"5e1fc1cd-109e-4bb8-aa37-19c71590b563","Type":"ContainerStarted","Data":"853970c80bdb24c00e78a47dd8070c7c24d2626dbb0c2b0f27a6cd5009cb6c87"} Sep 18 16:15:55.736204 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.735199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9s65v" event={"ID":"81ce707b-f655-4f08-aed9-315baa4ac890","Type":"ContainerStarted","Data":"0f63ac64002b500ea8cfe2d98988267780e80545cc8882425e488ddf78d1a155"} Sep 18 16:15:55.742342 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.742225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xplqn" event={"ID":"17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4","Type":"ContainerStarted","Data":"96f99d1919ac673bc0be57e34d9840779afebe9a96964e84c8b055d7ac9c35ac"} Sep 18 16:15:55.743978 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.743955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" event={"ID":"06f6e7c9-8a1c-40fd-a2ba-643769658c5e","Type":"ContainerStarted","Data":"b006290a0b739ccc9e7ebbfae89279971ab713240f8c8563708398dd540f8a11"} Sep 18 16:15:55.745534 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.745467 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8vqrx" podStartSLOduration=3.870054143 podStartE2EDuration="20.745452339s" totalImagesPullingTime="16.875398196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:55.745373473 +0000 UTC m=+22.264640984" watchObservedRunningTime="2026-09-18 16:15:55.745452339 +0000 UTC m=+22.264719839" Sep 18 16:15:55.745647 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.745608 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-79.ec2.internal" podStartSLOduration=19.745601199 podStartE2EDuration="19.745601199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:40.717837144 +0000 UTC m=+7.237104643" watchObservedRunningTime="2026-09-18 16:15:55.745601199 +0000 UTC m=+22.264868699" Sep 18 16:15:55.746043 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.746022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cffmd" event={"ID":"4ae800b0-36f5-4c25-b07c-cd35de96f0d8","Type":"ContainerStarted","Data":"6e5a5e86651b50169dfa3748e0991e109e862b2afd002dfa4714f02ab6b81622"} Sep 18 16:15:55.773399 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.773366 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wztrg" podStartSLOduration=3.895198893 podStartE2EDuration="20.773354259s" totalImagesPullingTime="16.878155366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:55.759495733 +0000 UTC m=+22.278763224" watchObservedRunningTime="2026-09-18 16:15:55.773354259 +0000 UTC m=+22.292621758" Sep 18 16:15:55.773539 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.773519 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xplqn" podStartSLOduration=6.958131628 podStartE2EDuration="19.773514101s" totalImagesPullingTime="12.815382473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:55.773475746 +0000 UTC m=+22.292743228" watchObservedRunningTime="2026-09-18 16:15:55.773514101 +0000 UTC m=+22.292781596" Sep 18 16:15:55.793194 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:55.793163 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9s65v" podStartSLOduration=3.87152796 podStartE2EDuration="20.793153234s" totalImagesPullingTime="16.921625274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:55.792848561 +0000 UTC m=+22.312116060" watchObservedRunningTime="2026-09-18 16:15:55.793153234 +0000 UTC m=+22.312420731" Sep 18 16:15:56.634658 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.634630 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:56.634780 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:56.634728 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:56.749795 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.749711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wkmhp" event={"ID":"fead61ad-011c-4b2a-81f1-0967e1811606","Type":"ContainerStarted","Data":"3d5f2db489e1e6ae1faf3fd288cf531a4492039fba14dd31806fdfe123501cec"} Sep 18 16:15:56.751156 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.751130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f2vz7" event={"ID":"931320fc-4a72-46a3-94f2-431ff1c66526","Type":"ContainerStarted","Data":"dfe24486ec18c989bcdf9d8f5d7e6a269a6a12e1a5b230905304d7df47c38ae4"} Sep 18 16:15:56.752441 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.752420 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ae800b0-36f5-4c25-b07c-cd35de96f0d8" containerID="6e5a5e86651b50169dfa3748e0991e109e862b2afd002dfa4714f02ab6b81622" exitCode=0 Sep 18 16:15:56.752535 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.752511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cffmd" event={"ID":"4ae800b0-36f5-4c25-b07c-cd35de96f0d8","Type":"ContainerDied","Data":"6e5a5e86651b50169dfa3748e0991e109e862b2afd002dfa4714f02ab6b81622"} Sep 18 16:15:56.755178 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.755159 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:15:56.755516 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.755480 2564 generic.go:356] "Generic (PLEG): container finished" podID="45cc0900-9bb9-4d4e-9ca6-7f19aa39807f" containerID="782a22404a02f5401a0589d2291e1614e84696fa21ee6b1a6d83e07aa10ff00b" exitCode=1 Sep 18 16:15:56.755605 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.755514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" event={"ID":"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f","Type":"ContainerStarted","Data":"022c2cae86d68914a6db1357a95d1e70eef08255fb195f21640bdcad9bd2c26a"} Sep 18 16:15:56.755605 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.755591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" event={"ID":"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f","Type":"ContainerStarted","Data":"8c110369c0f2ad1e94196e2feec7a1f72952c08db3366640a34584350f527efe"} Sep 18 16:15:56.755702 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.755645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" event={"ID":"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f","Type":"ContainerStarted","Data":"779648efd2d7165d511c0897d9891cf14bab730adbc590ac8f8d9cad82e8c13e"} Sep 18 16:15:56.755702 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.755692 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" event={"ID":"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f","Type":"ContainerStarted","Data":"025a5675bd0670e059800af6dad4282c5c30996291eadfeb9bbebf8ebc024a94"} Sep 18 16:15:56.755797 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.755738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" event={"ID":"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f","Type":"ContainerDied","Data":"782a22404a02f5401a0589d2291e1614e84696fa21ee6b1a6d83e07aa10ff00b"} Sep 18 16:15:56.755797 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.755786 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" event={"ID":"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f","Type":"ContainerStarted","Data":"3a5e532c4f50daa38af61030d3bef4d7c407304829a46b6ec88e8416fe9a81a3"} Sep 18 16:15:56.765321 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.765288 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wkmhp" podStartSLOduration=4.85439314 podStartE2EDuration="21.765276833s" totalImagesPullingTime="16.910883693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:56.764920392 +0000 UTC m=+23.284187891" watchObservedRunningTime="2026-09-18 16:15:56.765276833 +0000 UTC m=+23.284544332" Sep 18 16:15:56.807085 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.807042 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f2vz7" podStartSLOduration=3.840097924 podStartE2EDuration="20.807027348s" totalImagesPullingTime="16.966929424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:56.806993663 +0000 UTC m=+23.326261162" watchObservedRunningTime="2026-09-18 16:15:56.807027348 +0000 UTC m=+23.326294890" Sep 18 16:15:56.964474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.964441 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:56.965152 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:56.965125 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:57.047543 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:57.047521 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 18 16:15:57.579448 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:57.579349 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:15:57.047537198Z","UUID":"fe1f81fc-e8d5-4253-abd4-4fcc0c6089bf","Handler":null,"Name":"","Endpoint":""} Sep 18 16:15:57.581151 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:57.581126 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 18 16:15:57.581273 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:57.581162 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 18 16:15:57.634498 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:57.634452 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:57.634690 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:57.634602 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:57.634941 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:57.634452 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:57.635054 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:57.635032 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:15:57.761196 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:57.761152 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" event={"ID":"06f6e7c9-8a1c-40fd-a2ba-643769658c5e","Type":"ContainerStarted","Data":"0d5670eb9d1b70d3cb164804f3f2e5401928104f53becf17e83e94e1d4c0539c"} Sep 18 16:15:57.761698 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:57.761651 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:57.761918 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:57.761898 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xplqn" Sep 18 16:15:58.634835 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:58.634680 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:15:58.634988 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:58.634907 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:15:58.764821 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:58.764793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" event={"ID":"06f6e7c9-8a1c-40fd-a2ba-643769658c5e","Type":"ContainerStarted","Data":"01811cb2395a444d2b80ed0a302e99892ab9145f5a31ce644e0ca8e363d03537"} Sep 18 16:15:58.767698 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:58.767682 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:15:58.768018 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:58.768000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" event={"ID":"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f","Type":"ContainerStarted","Data":"a5a5f0ce3c8f6fc3f7b4cc57735263e5bea9bfac0ffe909c2178e63ca91ab187"} Sep 18 16:15:58.785357 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:58.785299 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5bd" podStartSLOduration=4.632899948 podStartE2EDuration="23.78528238s" totalImagesPullingTime="19.152382432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:58.784652464 +0000 UTC m=+25.303919962" watchObservedRunningTime="2026-09-18 16:15:58.78528238 +0000 UTC m=+25.304549880" Sep 18 16:15:59.634898 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:59.634867 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:15:59.635098 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:15:59.634868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:15:59.635098 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:59.635002 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:15:59.635098 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:15:59.635048 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:16:00.634859 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:00.634830 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:16:00.635356 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:00.634950 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:16:01.635099 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.635070 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:16:01.635477 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:01.635188 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:16:01.635477 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.635234 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:16:01.635477 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:01.635325 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:16:01.775707 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.775677 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cffmd" event={"ID":"4ae800b0-36f5-4c25-b07c-cd35de96f0d8","Type":"ContainerStarted","Data":"880000536bc9d0cbdc1d96710cb02e55accd14cbc3c69018760cf4be05449be4"} Sep 18 16:16:01.779911 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.779713 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:16:01.780317 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.780293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" event={"ID":"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f","Type":"ContainerStarted","Data":"51177a680838c1222d85bb2677c0fb12964f65cf1f306147ca3bc1c72cbd2df2"} Sep 18 16:16:01.780609 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.780588 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:16:01.780705 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.780659 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:16:01.780755 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.780711 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:16:01.780859 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.780841 2564 scope.go:117] "RemoveContainer" containerID="782a22404a02f5401a0589d2291e1614e84696fa21ee6b1a6d83e07aa10ff00b" Sep 18 16:16:01.797600 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.797574 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:16:01.799279 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:01.799260 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:16:02.634374 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:02.634352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:16:02.634465 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:02.634448 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:16:02.784467 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:02.784440 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ae800b0-36f5-4c25-b07c-cd35de96f0d8" containerID="880000536bc9d0cbdc1d96710cb02e55accd14cbc3c69018760cf4be05449be4" exitCode=0 Sep 18 16:16:02.784963 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:02.784522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cffmd" event={"ID":"4ae800b0-36f5-4c25-b07c-cd35de96f0d8","Type":"ContainerDied","Data":"880000536bc9d0cbdc1d96710cb02e55accd14cbc3c69018760cf4be05449be4"} Sep 18 16:16:02.788104 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:02.788089 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:16:02.788428 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:02.788407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" event={"ID":"45cc0900-9bb9-4d4e-9ca6-7f19aa39807f","Type":"ContainerStarted","Data":"d371cd1e977cdbb1dfbdf086a4ccba056d2f8be6ac418eb6535df74ee8b906c9"} Sep 18 16:16:03.577428 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:03.577381 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" podStartSLOduration=11.652843514 podStartE2EDuration="28.577364731s" totalImagesPullingTime="16.924521217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:02.839310876 +0000 UTC m=+29.358578374" watchObservedRunningTime="2026-09-18 16:16:03.577364731 +0000 UTC m=+30.096632228" Sep 18 16:16:03.577654 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:03.577629 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-msncf"] Sep 18 16:16:03.577776 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:03.577724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:16:03.577835 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:03.577803 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:16:03.581409 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:03.581387 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7xjxp"] Sep 18 16:16:03.581531 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:03.581504 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:16:03.581628 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:03.581607 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:16:03.581970 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:03.581950 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r4pj7"] Sep 18 16:16:03.582041 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:03.582030 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:16:03.582133 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:03.582113 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:16:03.792433 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:03.792399 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ae800b0-36f5-4c25-b07c-cd35de96f0d8" containerID="74f6fd226c0093249841157bebb21184007181eb5ccae2c392bbf23c0888c41a" exitCode=0 Sep 18 16:16:03.792874 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:03.792478 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cffmd" event={"ID":"4ae800b0-36f5-4c25-b07c-cd35de96f0d8","Type":"ContainerDied","Data":"74f6fd226c0093249841157bebb21184007181eb5ccae2c392bbf23c0888c41a"} Sep 18 16:16:04.634930 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:04.634898 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:16:04.635069 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:04.635011 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:16:05.635075 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:05.634895 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:16:05.635390 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:05.635092 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:16:05.635390 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:05.634967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:16:05.635390 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:05.635149 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:16:05.798744 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:05.798716 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ae800b0-36f5-4c25-b07c-cd35de96f0d8" containerID="e6915c622d1aa992cf87f67f47152b70aae4d6a90b2386ebc59a08f80964323e" exitCode=0 Sep 18 16:16:05.798852 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:05.798769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cffmd" event={"ID":"4ae800b0-36f5-4c25-b07c-cd35de96f0d8","Type":"ContainerDied","Data":"e6915c622d1aa992cf87f67f47152b70aae4d6a90b2386ebc59a08f80964323e"} Sep 18 16:16:06.634405 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:06.634373 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:16:06.634597 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:06.634467 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xjxp" podUID="02dcbb62-fa0e-491c-9428-59eb7856d6b2" Sep 18 16:16:07.635057 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:07.635027 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:16:07.635465 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:07.635157 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4pj7" podUID="415ad465-e1a1-4c60-99dd-7c18748d64e8" Sep 18 16:16:07.635465 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:07.635226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:16:07.635465 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:07.635338 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msncf" podUID="458d51a9-e012-413e-8dd6-8354a56b8abc" Sep 18 16:16:08.222960 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.222934 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-79.ec2.internal" event="NodeReady" Sep 18 16:16:08.223082 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.223037 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 18 16:16:08.270403 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.270370 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86df76cfff-rzhm6"] Sep 18 16:16:08.272469 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.272451 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.274942 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.274906 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 18 16:16:08.275261 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.275065 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 18 16:16:08.277904 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.276588 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 18 16:16:08.277904 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.276713 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h482k\"" Sep 18 16:16:08.281403 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.279522 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4"] Sep 18 16:16:08.282698 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.282674 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd"] Sep 18 16:16:08.284050 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.283642 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.284050 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.283989 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 18 16:16:08.284676 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.284658 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-6zxcq"] Sep 18 16:16:08.284788 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.284772 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:08.286362 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.286341 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-mbpfm"] Sep 18 16:16:08.286476 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.286456 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.287170 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.287153 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:16:08.287294 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.287276 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 18 16:16:08.287394 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.287350 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 18 16:16:08.287465 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.287411 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 18 16:16:08.287531 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.287476 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 18 16:16:08.287581 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.287551 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zf9wk\"" Sep 18 16:16:08.287581 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.287566 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 18 16:16:08.287654 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.287635 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 18 16:16:08.287792 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.287776 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cxft6\"" Sep 18 16:16:08.288652 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.288631 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-787796f9d4-z4dch"] Sep 18 16:16:08.288778 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.288705 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 18 16:16:08.289051 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.288999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.290600 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.290578 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d6rcp\"" Sep 18 16:16:08.290703 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.290633 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g"] Sep 18 16:16:08.290836 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.290817 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.290917 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.290636 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 18 16:16:08.291526 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.291508 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 18 16:16:08.291612 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.291527 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 18 16:16:08.291963 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.291946 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 18 16:16:08.292061 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.291989 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:16:08.292146 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.292062 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 18 16:16:08.292204 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.292154 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 18 16:16:08.292204 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.292158 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-h2h2p\"" Sep 18 16:16:08.292313 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.292238 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 18 16:16:08.295630 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.295436 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 18 16:16:08.295630 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.295479 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 18 16:16:08.295630 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.295547 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 18 16:16:08.295630 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.295479 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 18 16:16:08.295836 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.295632 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rjpnh\"" Sep 18 16:16:08.295836 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.295716 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 18 16:16:08.295836 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.295782 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9"] Sep 18 16:16:08.295980 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.295882 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 18 16:16:08.296029 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.295997 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g" Sep 18 16:16:08.297672 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.297643 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86df76cfff-rzhm6"] Sep 18 16:16:08.298076 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.298058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:08.299093 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.299074 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 18 16:16:08.299171 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.299115 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:16:08.299783 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.299733 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-92smk"] Sep 18 16:16:08.300349 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.300330 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dj9ts\"" Sep 18 16:16:08.301695 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.301674 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hm75f"] Sep 18 16:16:08.301802 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.301786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-92smk" Sep 18 16:16:08.304228 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.304209 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.304959 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.304938 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 18 16:16:08.309435 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.309412 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5rtvs\"" Sep 18 16:16:08.309798 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.309666 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 18 16:16:08.309957 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.309939 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 18 16:16:08.310159 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.310138 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 18 16:16:08.310260 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.310244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 18 16:16:08.310502 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.310472 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 18 16:16:08.310649 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.310630 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 18 16:16:08.310880 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.310861 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:16:08.310991 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.310973 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nst46\"" Sep 18 16:16:08.311383 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.311362 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-44fnz\"" Sep 18 16:16:08.311884 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.311865 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 18 16:16:08.312295 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.312272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4"] Sep 18 16:16:08.313357 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.312684 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 18 16:16:08.313865 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.313842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd"] Sep 18 16:16:08.315832 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.315722 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g"] Sep 18 16:16:08.316904 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.316889 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-6zxcq"] Sep 18 16:16:08.317688 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.317672 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9"] Sep 18 16:16:08.318605 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.318541 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-787796f9d4-z4dch"] Sep 18 16:16:08.319411 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.319390 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-92smk"] Sep 18 16:16:08.324577 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.324557 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-mbpfm"] Sep 18 16:16:08.335613 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.335595 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hm75f"] Sep 18 16:16:08.433805 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.433781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fh9q\" (UniqueName: \"kubernetes.io/projected/9c9fda39-70e8-4d3b-9e16-d767847f72e0-kube-api-access-8fh9q\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:08.434026 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.433809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-stats-auth\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.434026 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.433833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0047f8a-de97-43ae-b9b5-679de1b69eef-serving-cert\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.434026 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.433888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.434026 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.433928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-image-registry-private-configuration\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.434026 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.433952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.434026 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.433969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85cp\" (UniqueName: \"kubernetes.io/projected/52b086e1-8d64-4d18-93c6-f411c3cb7b87-kube-api-access-q85cp\") pod \"volume-data-source-validator-5c4db8bdf9-c5s8g\" (UID: \"52b086e1-8d64-4d18-93c6-f411c3cb7b87\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g" Sep 18 16:16:08.434280 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434037 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6ghs\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-kube-api-access-m6ghs\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.434280 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.434280 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztrv\" (UniqueName: \"kubernetes.io/projected/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-kube-api-access-fztrv\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.434280 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-config-volume\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.434280 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b2e70b6-bc6d-441d-8447-91ace8d34925-ca-trust-extracted\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.434280 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql2dc\" (UniqueName: \"kubernetes.io/projected/73610849-9bcc-4084-a60e-a0ee6a2a7861-kube-api-access-ql2dc\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.434280 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-tmp-dir\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.434280 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-bound-sa-token\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9c9fda39-70e8-4d3b-9e16-d767847f72e0-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bt6w\" (UniqueName: \"kubernetes.io/projected/f1b73e02-8e1c-41db-a213-b8baee6c4e54-kube-api-access-9bt6w\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0047f8a-de97-43ae-b9b5-679de1b69eef-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-wbrs4\" (UID: \"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-default-certificate\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d14e4595-8b55-4f60-a557-aa750c699157-config\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp2k\" (UniqueName: \"kubernetes.io/projected/d14e4595-8b55-4f60-a557-aa750c699157-kube-api-access-cvp2k\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d14e4595-8b55-4f60-a557-aa750c699157-trusted-ca\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87108184-50a2-4e70-9887-69a7b01af53f-cert\") pod \"ingress-canary-92smk\" (UID: \"87108184-50a2-4e70-9887-69a7b01af53f\") " pod="openshift-ingress-canary/ingress-canary-92smk" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.434628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434608 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0047f8a-de97-43ae-b9b5-679de1b69eef-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72pg\" (UniqueName: \"kubernetes.io/projected/b0047f8a-de97-43ae-b9b5-679de1b69eef-kube-api-access-w72pg\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skf9\" (UniqueName: \"kubernetes.io/projected/87108184-50a2-4e70-9887-69a7b01af53f-kube-api-access-4skf9\") pod \"ingress-canary-92smk\" (UID: \"87108184-50a2-4e70-9887-69a7b01af53f\") " pod="openshift-ingress-canary/ingress-canary-92smk" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-trusted-ca\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434743 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-installation-pull-secrets\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0047f8a-de97-43ae-b9b5-679de1b69eef-tmp\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d14e4595-8b55-4f60-a557-aa750c699157-serving-cert\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhnjp\" (UniqueName: \"kubernetes.io/projected/bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af-kube-api-access-qhnjp\") pod \"kube-storage-version-migrator-operator-55889f7b47-wbrs4\" (UID: \"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434893 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-certificates\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0047f8a-de97-43ae-b9b5-679de1b69eef-snapshots\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.435127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.434993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-wbrs4\" (UID: \"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.535476 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-tmp-dir\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.535566 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-bound-sa-token\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.535566 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9c9fda39-70e8-4d3b-9e16-d767847f72e0-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:08.535566 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bt6w\" (UniqueName: \"kubernetes.io/projected/f1b73e02-8e1c-41db-a213-b8baee6c4e54-kube-api-access-9bt6w\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:08.535689 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0047f8a-de97-43ae-b9b5-679de1b69eef-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.535689 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-wbrs4\" (UID: \"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.535761 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535689 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-default-certificate\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.535761 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d14e4595-8b55-4f60-a557-aa750c699157-config\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.535761 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp2k\" (UniqueName: \"kubernetes.io/projected/d14e4595-8b55-4f60-a557-aa750c699157-kube-api-access-cvp2k\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.535864 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d14e4595-8b55-4f60-a557-aa750c699157-trusted-ca\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.535864 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87108184-50a2-4e70-9887-69a7b01af53f-cert\") pod \"ingress-canary-92smk\" (UID: \"87108184-50a2-4e70-9887-69a7b01af53f\") " pod="openshift-ingress-canary/ingress-canary-92smk" Sep 18 16:16:08.535864 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-tmp-dir\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.535864 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.535864 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0047f8a-de97-43ae-b9b5-679de1b69eef-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.536056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72pg\" (UniqueName: \"kubernetes.io/projected/b0047f8a-de97-43ae-b9b5-679de1b69eef-kube-api-access-w72pg\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.536056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:08.536056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skf9\" (UniqueName: \"kubernetes.io/projected/87108184-50a2-4e70-9887-69a7b01af53f-kube-api-access-4skf9\") pod \"ingress-canary-92smk\" (UID: \"87108184-50a2-4e70-9887-69a7b01af53f\") " pod="openshift-ingress-canary/ingress-canary-92smk" Sep 18 16:16:08.536056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-trusted-ca\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.536056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.535979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-installation-pull-secrets\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.536056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:08.536056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0047f8a-de97-43ae-b9b5-679de1b69eef-tmp\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d14e4595-8b55-4f60-a557-aa750c699157-serving-cert\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhnjp\" (UniqueName: \"kubernetes.io/projected/bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af-kube-api-access-qhnjp\") pod \"kube-storage-version-migrator-operator-55889f7b47-wbrs4\" (UID: \"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-certificates\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0047f8a-de97-43ae-b9b5-679de1b69eef-snapshots\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-wbrs4\" (UID: \"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fh9q\" (UniqueName: \"kubernetes.io/projected/9c9fda39-70e8-4d3b-9e16-d767847f72e0-kube-api-access-8fh9q\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-stats-auth\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0047f8a-de97-43ae-b9b5-679de1b69eef-serving-cert\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-image-registry-private-configuration\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.536348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.536835 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85cp\" (UniqueName: \"kubernetes.io/projected/52b086e1-8d64-4d18-93c6-f411c3cb7b87-kube-api-access-q85cp\") pod \"volume-data-source-validator-5c4db8bdf9-c5s8g\" (UID: \"52b086e1-8d64-4d18-93c6-f411c3cb7b87\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g" Sep 18 16:16:08.536835 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6ghs\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-kube-api-access-m6ghs\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.536835 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536400 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0047f8a-de97-43ae-b9b5-679de1b69eef-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.536835 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.536835 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztrv\" (UniqueName: \"kubernetes.io/projected/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-kube-api-access-fztrv\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.536835 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536471 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d14e4595-8b55-4f60-a557-aa750c699157-config\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.536835 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536635 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d14e4595-8b55-4f60-a557-aa750c699157-trusted-ca\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.537134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-config-volume\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.537134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.536477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-config-volume\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.537134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.537022 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-wbrs4\" (UID: \"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.537134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.537044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b2e70b6-bc6d-441d-8447-91ace8d34925-ca-trust-extracted\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.537134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.537073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql2dc\" (UniqueName: \"kubernetes.io/projected/73610849-9bcc-4084-a60e-a0ee6a2a7861-kube-api-access-ql2dc\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.537134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.537084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0047f8a-de97-43ae-b9b5-679de1b69eef-snapshots\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.538013 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.537896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0047f8a-de97-43ae-b9b5-679de1b69eef-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.538319 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.538263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9c9fda39-70e8-4d3b-9e16-d767847f72e0-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:08.538414 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.537077 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:16:08.538414 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.538380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-trusted-ca\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.538558 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.538427 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls podName:f1b73e02-8e1c-41db-a213-b8baee6c4e54 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.038406607 +0000 UTC m=+35.557674096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dxrt9" (UID: "f1b73e02-8e1c-41db-a213-b8baee6c4e54") : secret "samples-operator-tls" not found Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.537107 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.538657 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86df76cfff-rzhm6: secret "image-registry-tls" not found Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.538656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b2e70b6-bc6d-441d-8447-91ace8d34925-ca-trust-extracted\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.537250 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.538704 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls podName:3b2e70b6-bc6d-441d-8447-91ace8d34925 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.038689222 +0000 UTC m=+35.557956714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls") pod "image-registry-86df76cfff-rzhm6" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925") : secret "image-registry-tls" not found Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.538741 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls podName:9c9fda39-70e8-4d3b-9e16-d767847f72e0 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.03872345 +0000 UTC m=+35.557990945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4blfd" (UID: "9c9fda39-70e8-4d3b-9e16-d767847f72e0") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.537277 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.538760 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.038749501 +0000 UTC m=+35.558016985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : configmap references non-existent config key: service-ca.crt Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.538784 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls podName:81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.038773154 +0000 UTC m=+35.558040660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls") pod "dns-default-hm75f" (UID: "81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e") : secret "dns-default-metrics-tls" not found Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.537970 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:08.538817 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.038809244 +0000 UTC m=+35.558076748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : secret "router-metrics-certs-default" not found Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.539183 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0047f8a-de97-43ae-b9b5-679de1b69eef-tmp\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.539384 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.539257 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-certificates\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.540898 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.540879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-wbrs4\" (UID: \"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.541195 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.541170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-default-certificate\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.541372 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.541351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-image-registry-private-configuration\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.541445 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.541392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-installation-pull-secrets\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.541871 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.541848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87108184-50a2-4e70-9887-69a7b01af53f-cert\") pod \"ingress-canary-92smk\" (UID: \"87108184-50a2-4e70-9887-69a7b01af53f\") " pod="openshift-ingress-canary/ingress-canary-92smk" Sep 18 16:16:08.541995 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.541977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-stats-auth\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.542345 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.542326 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d14e4595-8b55-4f60-a557-aa750c699157-serving-cert\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.542874 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.542852 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0047f8a-de97-43ae-b9b5-679de1b69eef-serving-cert\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.546150 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.546127 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-bound-sa-token\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.552138 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.552105 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp2k\" (UniqueName: \"kubernetes.io/projected/d14e4595-8b55-4f60-a557-aa750c699157-kube-api-access-cvp2k\") pod \"console-operator-fd4bb46c-mbpfm\" (UID: \"d14e4595-8b55-4f60-a557-aa750c699157\") " pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.552395 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.552351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql2dc\" (UniqueName: \"kubernetes.io/projected/73610849-9bcc-4084-a60e-a0ee6a2a7861-kube-api-access-ql2dc\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:08.553340 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.553316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bt6w\" (UniqueName: \"kubernetes.io/projected/f1b73e02-8e1c-41db-a213-b8baee6c4e54-kube-api-access-9bt6w\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:08.553447 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.553432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fh9q\" (UniqueName: \"kubernetes.io/projected/9c9fda39-70e8-4d3b-9e16-d767847f72e0-kube-api-access-8fh9q\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:08.553989 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.553950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72pg\" (UniqueName: \"kubernetes.io/projected/b0047f8a-de97-43ae-b9b5-679de1b69eef-kube-api-access-w72pg\") pod \"insights-operator-6f9c644dd9-6zxcq\" (UID: \"b0047f8a-de97-43ae-b9b5-679de1b69eef\") " pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.554078 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.554010 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85cp\" (UniqueName: \"kubernetes.io/projected/52b086e1-8d64-4d18-93c6-f411c3cb7b87-kube-api-access-q85cp\") pod \"volume-data-source-validator-5c4db8bdf9-c5s8g\" (UID: \"52b086e1-8d64-4d18-93c6-f411c3cb7b87\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g" Sep 18 16:16:08.554078 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.554023 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztrv\" (UniqueName: \"kubernetes.io/projected/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-kube-api-access-fztrv\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:08.555202 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.555182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skf9\" (UniqueName: \"kubernetes.io/projected/87108184-50a2-4e70-9887-69a7b01af53f-kube-api-access-4skf9\") pod \"ingress-canary-92smk\" (UID: \"87108184-50a2-4e70-9887-69a7b01af53f\") " pod="openshift-ingress-canary/ingress-canary-92smk" Sep 18 16:16:08.555312 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.555222 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhnjp\" (UniqueName: \"kubernetes.io/projected/bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af-kube-api-access-qhnjp\") pod \"kube-storage-version-migrator-operator-55889f7b47-wbrs4\" (UID: \"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.555312 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.555300 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6ghs\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-kube-api-access-m6ghs\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:08.598593 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.598572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" Sep 18 16:16:08.601388 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.601369 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv"] Sep 18 16:16:08.603348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.603331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv" Sep 18 16:16:08.606087 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.606061 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 16:16:08.606176 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.606104 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 16:16:08.606673 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.606656 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4mfcs\"" Sep 18 16:16:08.619458 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.619096 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" Sep 18 16:16:08.623383 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.623265 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv"] Sep 18 16:16:08.627871 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.627783 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:08.634097 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.634055 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:16:08.637268 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.637046 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-42l89\"" Sep 18 16:16:08.640641 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.640371 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g" Sep 18 16:16:08.655191 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.655167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-92smk" Sep 18 16:16:08.741174 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.739072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kljsm\" (UniqueName: \"kubernetes.io/projected/38551f86-17da-4945-84d1-b8f8cccbcd73-kube-api-access-kljsm\") pod \"network-check-source-7cbc45fddc-z55rv\" (UID: \"38551f86-17da-4945-84d1-b8f8cccbcd73\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv" Sep 18 16:16:08.840110 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.840079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kljsm\" (UniqueName: \"kubernetes.io/projected/38551f86-17da-4945-84d1-b8f8cccbcd73-kube-api-access-kljsm\") pod \"network-check-source-7cbc45fddc-z55rv\" (UID: \"38551f86-17da-4945-84d1-b8f8cccbcd73\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv" Sep 18 16:16:08.849576 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.849552 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kljsm\" (UniqueName: \"kubernetes.io/projected/38551f86-17da-4945-84d1-b8f8cccbcd73-kube-api-access-kljsm\") pod \"network-check-source-7cbc45fddc-z55rv\" (UID: \"38551f86-17da-4945-84d1-b8f8cccbcd73\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv" Sep 18 16:16:08.892785 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.892760 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4"] Sep 18 16:16:08.895742 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:08.895718 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3cc907_5ce5_4d5e_89d9_3e60eeb0f1af.slice/crio-d87d3b87cce56ae6d0e1aebff918dc2b6a55e9935d1a32c642f3184f3b4fd84f WatchSource:0}: Error finding container d87d3b87cce56ae6d0e1aebff918dc2b6a55e9935d1a32c642f3184f3b4fd84f: Status 404 returned error can't find the container with id d87d3b87cce56ae6d0e1aebff918dc2b6a55e9935d1a32c642f3184f3b4fd84f Sep 18 16:16:08.924525 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.924479 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-6zxcq"] Sep 18 16:16:08.927350 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:08.927325 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0047f8a_de97_43ae_b9b5_679de1b69eef.slice/crio-0119202c22247fc6cd87a0cfc848e2f0f58c984b955c6202d573e3c1acfe0764 WatchSource:0}: Error finding container 0119202c22247fc6cd87a0cfc848e2f0f58c984b955c6202d573e3c1acfe0764: Status 404 returned error can't find the container with id 0119202c22247fc6cd87a0cfc848e2f0f58c984b955c6202d573e3c1acfe0764 Sep 18 16:16:08.927451 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.927386 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv" Sep 18 16:16:08.927451 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.927399 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-mbpfm"] Sep 18 16:16:08.930257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:08.930201 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14e4595_8b55_4f60_a557_aa750c699157.slice/crio-9d5d2eb424f73022aa3445a04239cc4475d29f00597af3f36e244b6e73229730 WatchSource:0}: Error finding container 9d5d2eb424f73022aa3445a04239cc4475d29f00597af3f36e244b6e73229730: Status 404 returned error can't find the container with id 9d5d2eb424f73022aa3445a04239cc4475d29f00597af3f36e244b6e73229730 Sep 18 16:16:08.952158 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.952035 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g"] Sep 18 16:16:08.953592 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:08.952998 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-92smk"] Sep 18 16:16:08.955679 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:08.955642 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b086e1_8d64_4d18_93c6_f411c3cb7b87.slice/crio-a1deabae32c187b372ab106eb045ab68f84533b8c24f7c33f7ab688b922655e7 WatchSource:0}: Error finding container a1deabae32c187b372ab106eb045ab68f84533b8c24f7c33f7ab688b922655e7: Status 404 returned error can't find the container with id a1deabae32c187b372ab106eb045ab68f84533b8c24f7c33f7ab688b922655e7 Sep 18 16:16:08.956127 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:08.956102 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87108184_50a2_4e70_9887_69a7b01af53f.slice/crio-ceb8b70325d0f95e5b8b85f103e50086bbd69352766d62619dea642e67509e33 WatchSource:0}: Error finding container ceb8b70325d0f95e5b8b85f103e50086bbd69352766d62619dea642e67509e33: Status 404 returned error can't find the container with id ceb8b70325d0f95e5b8b85f103e50086bbd69352766d62619dea642e67509e33 Sep 18 16:16:09.041982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.041947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:09.042134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.042002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:09.042134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.042048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:09.042134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.042108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:09.042293 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042137 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:10.042112739 +0000 UTC m=+36.561380236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : configmap references non-existent config key: service-ca.crt Sep 18 16:16:09.042293 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042174 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:16:09.042293 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042189 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:16:09.042293 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042195 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86df76cfff-rzhm6: secret "image-registry-tls" not found Sep 18 16:16:09.042293 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042238 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls podName:81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:10.042222747 +0000 UTC m=+36.561490326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls") pod "dns-default-hm75f" (UID: "81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e") : secret "dns-default-metrics-tls" not found Sep 18 16:16:09.042293 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042239 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:16:09.042293 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.042184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:09.042293 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042255 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls podName:3b2e70b6-bc6d-441d-8447-91ace8d34925 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:10.042246343 +0000 UTC m=+36.561513821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls") pod "image-registry-86df76cfff-rzhm6" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925") : secret "image-registry-tls" not found Sep 18 16:16:09.042293 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042239 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:16:09.042293 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042290 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls podName:f1b73e02-8e1c-41db-a213-b8baee6c4e54 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:10.042271585 +0000 UTC m=+36.561539073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dxrt9" (UID: "f1b73e02-8e1c-41db-a213-b8baee6c4e54") : secret "samples-operator-tls" not found Sep 18 16:16:09.042801 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042305 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:10.042298974 +0000 UTC m=+36.561566455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : secret "router-metrics-certs-default" not found Sep 18 16:16:09.042801 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.042349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:09.042801 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042463 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:09.042801 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.042525 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls podName:9c9fda39-70e8-4d3b-9e16-d767847f72e0 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:10.042511618 +0000 UTC m=+36.561779109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4blfd" (UID: "9c9fda39-70e8-4d3b-9e16-d767847f72e0") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:09.203989 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.203920 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv"] Sep 18 16:16:09.244427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.244400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:16:09.244583 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.244563 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:16:09.244642 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.244624 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs podName:415ad465-e1a1-4c60-99dd-7c18748d64e8 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:41.244609657 +0000 UTC m=+67.763877134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs") pod "network-metrics-daemon-r4pj7" (UID: "415ad465-e1a1-4c60-99dd-7c18748d64e8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:16:09.345261 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.345231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:16:09.345401 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.345319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkx\" (UniqueName: \"kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx\") pod \"network-check-target-7xjxp\" (UID: \"02dcbb62-fa0e-491c-9428-59eb7856d6b2\") " pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:16:09.345401 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.345381 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:16:09.345512 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:09.345436 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret podName:458d51a9-e012-413e-8dd6-8354a56b8abc nodeName:}" failed. No retries permitted until 2026-09-18 16:16:41.34542229 +0000 UTC m=+67.864689772 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret") pod "global-pull-secret-syncer-msncf" (UID: "458d51a9-e012-413e-8dd6-8354a56b8abc") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:16:09.348391 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.348367 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzgkx\" (UniqueName: \"kubernetes.io/projected/02dcbb62-fa0e-491c-9428-59eb7856d6b2-kube-api-access-mzgkx\") pod \"network-check-target-7xjxp\" (UID: \"02dcbb62-fa0e-491c-9428-59eb7856d6b2\") " pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:16:09.546585 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.546506 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:16:09.634318 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.634293 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:16:09.634515 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.634477 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:16:09.637133 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.637104 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kmlw\"" Sep 18 16:16:09.637133 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.637125 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 16:16:09.637283 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.637129 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 16:16:09.811513 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.811428 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-92smk" event={"ID":"87108184-50a2-4e70-9887-69a7b01af53f","Type":"ContainerStarted","Data":"ceb8b70325d0f95e5b8b85f103e50086bbd69352766d62619dea642e67509e33"} Sep 18 16:16:09.812699 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.812675 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" event={"ID":"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af","Type":"ContainerStarted","Data":"d87d3b87cce56ae6d0e1aebff918dc2b6a55e9935d1a32c642f3184f3b4fd84f"} Sep 18 16:16:09.813941 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.813905 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g" event={"ID":"52b086e1-8d64-4d18-93c6-f411c3cb7b87","Type":"ContainerStarted","Data":"a1deabae32c187b372ab106eb045ab68f84533b8c24f7c33f7ab688b922655e7"} Sep 18 16:16:09.815204 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.815181 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" event={"ID":"b0047f8a-de97-43ae-b9b5-679de1b69eef","Type":"ContainerStarted","Data":"0119202c22247fc6cd87a0cfc848e2f0f58c984b955c6202d573e3c1acfe0764"} Sep 18 16:16:09.816442 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:09.816410 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" event={"ID":"d14e4595-8b55-4f60-a557-aa750c699157","Type":"ContainerStarted","Data":"9d5d2eb424f73022aa3445a04239cc4475d29f00597af3f36e244b6e73229730"} Sep 18 16:16:10.051533 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.051500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:10.051706 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.051544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:10.051706 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.051575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:10.051706 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.051627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:10.051706 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.051653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:10.051706 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051680 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051745 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051759 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051772 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86df76cfff-rzhm6: secret "image-registry-tls" not found Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051749 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls podName:81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.051730634 +0000 UTC m=+38.570998133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls") pod "dns-default-hm75f" (UID: "81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e") : secret "dns-default-metrics-tls" not found Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051796 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051680 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051825 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.051807802 +0000 UTC m=+38.571075281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : configmap references non-existent config key: service-ca.crt Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.051683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051847 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls podName:9c9fda39-70e8-4d3b-9e16-d767847f72e0 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.051834049 +0000 UTC m=+38.571101527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4blfd" (UID: "9c9fda39-70e8-4d3b-9e16-d767847f72e0") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051864 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls podName:3b2e70b6-bc6d-441d-8447-91ace8d34925 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.051857805 +0000 UTC m=+38.571125283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls") pod "image-registry-86df76cfff-rzhm6" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925") : secret "image-registry-tls" not found Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051887 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.051873118 +0000 UTC m=+38.571140595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : secret "router-metrics-certs-default" not found Sep 18 16:16:10.051953 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.051906 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls podName:f1b73e02-8e1c-41db-a213-b8baee6c4e54 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.051897025 +0000 UTC m=+38.571164508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dxrt9" (UID: "f1b73e02-8e1c-41db-a213-b8baee6c4e54") : secret "samples-operator-tls" not found Sep 18 16:16:10.312030 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.311999 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n"] Sep 18 16:16:10.314187 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.314167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.316398 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.316375 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:16:10.316943 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.316919 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 18 16:16:10.317121 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.317105 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 18 16:16:10.321313 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.321293 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 18 16:16:10.321440 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.321387 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2km2w\"" Sep 18 16:16:10.322222 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.322199 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n"] Sep 18 16:16:10.454761 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.454728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9nn\" (UniqueName: \"kubernetes.io/projected/ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f-kube-api-access-5f9nn\") pod \"service-ca-operator-57cdc4b768-d4c7n\" (UID: \"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.454860 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.454835 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f-serving-cert\") pod \"service-ca-operator-57cdc4b768-d4c7n\" (UID: \"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.454920 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.454870 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f-config\") pod \"service-ca-operator-57cdc4b768-d4c7n\" (UID: \"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.555988 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.555921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9nn\" (UniqueName: \"kubernetes.io/projected/ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f-kube-api-access-5f9nn\") pod \"service-ca-operator-57cdc4b768-d4c7n\" (UID: \"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.556106 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.556015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f-serving-cert\") pod \"service-ca-operator-57cdc4b768-d4c7n\" (UID: \"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.556106 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.556042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f-config\") pod \"service-ca-operator-57cdc4b768-d4c7n\" (UID: \"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.556844 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.556821 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f-config\") pod \"service-ca-operator-57cdc4b768-d4c7n\" (UID: \"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.558867 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.558831 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f-serving-cert\") pod \"service-ca-operator-57cdc4b768-d4c7n\" (UID: \"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.564308 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.564283 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9nn\" (UniqueName: \"kubernetes.io/projected/ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f-kube-api-access-5f9nn\") pod \"service-ca-operator-57cdc4b768-d4c7n\" (UID: \"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.630132 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.629952 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" Sep 18 16:16:10.654091 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.654062 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx"] Sep 18 16:16:10.661061 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.661037 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:10.663696 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.663560 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 18 16:16:10.663696 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.663611 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx"] Sep 18 16:16:10.666084 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.663917 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 18 16:16:10.666084 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.664153 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-m9mjg\"" Sep 18 16:16:10.725936 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.724344 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7xjxp"] Sep 18 16:16:10.758574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.758461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/584c5c9c-2e21-4c1f-a309-67e12d7d592c-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:10.759063 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.758852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:10.776849 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:10.776808 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dcbb62_fa0e_491c_9428_59eb7856d6b2.slice/crio-b0321ca88a97d12759168fc55028ab5b624fd2fd0186c2a307dc63fb4925dd00 WatchSource:0}: Error finding container b0321ca88a97d12759168fc55028ab5b624fd2fd0186c2a307dc63fb4925dd00: Status 404 returned error can't find the container with id b0321ca88a97d12759168fc55028ab5b624fd2fd0186c2a307dc63fb4925dd00 Sep 18 16:16:10.823108 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.823000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7xjxp" event={"ID":"02dcbb62-fa0e-491c-9428-59eb7856d6b2","Type":"ContainerStarted","Data":"b0321ca88a97d12759168fc55028ab5b624fd2fd0186c2a307dc63fb4925dd00"} Sep 18 16:16:10.824724 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.824649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv" event={"ID":"38551f86-17da-4945-84d1-b8f8cccbcd73","Type":"ContainerStarted","Data":"51f247dd398e65203480af0cd08bffb92277e45bb8ce892ea9985126350a2de5"} Sep 18 16:16:10.828575 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.828307 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ae800b0-36f5-4c25-b07c-cd35de96f0d8" containerID="8d0970c4cc7fc0d4336611aba61db6f47f8012a05376892cb2f0beea5ca9183a" exitCode=0 Sep 18 16:16:10.828575 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.828367 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cffmd" event={"ID":"4ae800b0-36f5-4c25-b07c-cd35de96f0d8","Type":"ContainerDied","Data":"8d0970c4cc7fc0d4336611aba61db6f47f8012a05376892cb2f0beea5ca9183a"} Sep 18 16:16:10.859741 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.859708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:10.860236 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.859866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/584c5c9c-2e21-4c1f-a309-67e12d7d592c-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:10.860236 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.859976 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:16:10.860236 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:10.860043 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert podName:584c5c9c-2e21-4c1f-a309-67e12d7d592c nodeName:}" failed. No retries permitted until 2026-09-18 16:16:11.36002332 +0000 UTC m=+37.879290811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ff5dx" (UID: "584c5c9c-2e21-4c1f-a309-67e12d7d592c") : secret "networking-console-plugin-cert" not found Sep 18 16:16:10.860603 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:10.860570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/584c5c9c-2e21-4c1f-a309-67e12d7d592c-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:11.018357 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:11.018281 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n"] Sep 18 16:16:11.022439 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:11.022405 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6e2edb_271b_43d0_89ee_d0e7a94f1c2f.slice/crio-2a67fe08edb818f6811676341b30a381e8b2d27a890b1a33d9be741d38645fd1 WatchSource:0}: Error finding container 2a67fe08edb818f6811676341b30a381e8b2d27a890b1a33d9be741d38645fd1: Status 404 returned error can't find the container with id 2a67fe08edb818f6811676341b30a381e8b2d27a890b1a33d9be741d38645fd1 Sep 18 16:16:11.363597 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:11.363277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:11.363779 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:11.363701 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:16:11.363779 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:11.363768 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert podName:584c5c9c-2e21-4c1f-a309-67e12d7d592c nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.363749276 +0000 UTC m=+38.883016761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ff5dx" (UID: "584c5c9c-2e21-4c1f-a309-67e12d7d592c") : secret "networking-console-plugin-cert" not found Sep 18 16:16:11.837991 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:11.837947 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" event={"ID":"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f","Type":"ContainerStarted","Data":"2a67fe08edb818f6811676341b30a381e8b2d27a890b1a33d9be741d38645fd1"} Sep 18 16:16:11.848423 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:11.847442 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ae800b0-36f5-4c25-b07c-cd35de96f0d8" containerID="8cf3dd624550aa993c827c35f679f415c3edebd5f691cc0b23aa1611e4def29c" exitCode=0 Sep 18 16:16:11.848423 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:11.847510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cffmd" event={"ID":"4ae800b0-36f5-4c25-b07c-cd35de96f0d8","Type":"ContainerDied","Data":"8cf3dd624550aa993c827c35f679f415c3edebd5f691cc0b23aa1611e4def29c"} Sep 18 16:16:12.078836 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:12.078802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:12.078956 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:12.078883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:12.078956 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:12.078920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:12.079077 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:12.079010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:12.079077 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:12.079038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:12.079077 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:12.079069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:12.079260 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.079222 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:12.079328 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.079284 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls podName:9c9fda39-70e8-4d3b-9e16-d767847f72e0 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:16.079266503 +0000 UTC m=+42.598533996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4blfd" (UID: "9c9fda39-70e8-4d3b-9e16-d767847f72e0") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:12.079726 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.079706 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:16.079689114 +0000 UTC m=+42.598956610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : configmap references non-existent config key: service-ca.crt Sep 18 16:16:12.079821 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.079789 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:16:12.079821 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.079801 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86df76cfff-rzhm6: secret "image-registry-tls" not found Sep 18 16:16:12.079920 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.079837 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls podName:3b2e70b6-bc6d-441d-8447-91ace8d34925 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:16.079824698 +0000 UTC m=+42.599092189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls") pod "image-registry-86df76cfff-rzhm6" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925") : secret "image-registry-tls" not found Sep 18 16:16:12.079920 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.079889 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:16:12.079920 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.079919 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:16.079908535 +0000 UTC m=+42.599176032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : secret "router-metrics-certs-default" not found Sep 18 16:16:12.080069 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.079969 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:16:12.080069 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.079994 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls podName:81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:16.079985646 +0000 UTC m=+42.599253138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls") pod "dns-default-hm75f" (UID: "81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e") : secret "dns-default-metrics-tls" not found Sep 18 16:16:12.080069 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.080042 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:16:12.080069 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.080070 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls podName:f1b73e02-8e1c-41db-a213-b8baee6c4e54 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:16.080060927 +0000 UTC m=+42.599328420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dxrt9" (UID: "f1b73e02-8e1c-41db-a213-b8baee6c4e54") : secret "samples-operator-tls" not found Sep 18 16:16:12.382920 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:12.382135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:12.382920 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.382348 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:16:12.382920 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:12.382413 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert podName:584c5c9c-2e21-4c1f-a309-67e12d7d592c nodeName:}" failed. No retries permitted until 2026-09-18 16:16:14.382395011 +0000 UTC m=+40.901662495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ff5dx" (UID: "584c5c9c-2e21-4c1f-a309-67e12d7d592c") : secret "networking-console-plugin-cert" not found Sep 18 16:16:12.857845 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:12.856968 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cffmd" event={"ID":"4ae800b0-36f5-4c25-b07c-cd35de96f0d8","Type":"ContainerStarted","Data":"b2caa0780f793be90ed933f0ccc2b4238d73c94eb0bda10b3d23bb38d35be1ad"} Sep 18 16:16:12.882135 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:12.881305 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cffmd" podStartSLOduration=8.916065077 podStartE2EDuration="36.881283687s" totalImagesPullingTime="27.96521861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:12.879683889 +0000 UTC m=+39.398951389" watchObservedRunningTime="2026-09-18 16:16:12.881283687 +0000 UTC m=+39.400551186" Sep 18 16:16:14.403947 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:14.403909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:14.404412 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:14.404031 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:16:14.404412 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:14.404083 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert podName:584c5c9c-2e21-4c1f-a309-67e12d7d592c nodeName:}" failed. No retries permitted until 2026-09-18 16:16:18.404070018 +0000 UTC m=+44.923337496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ff5dx" (UID: "584c5c9c-2e21-4c1f-a309-67e12d7d592c") : secret "networking-console-plugin-cert" not found Sep 18 16:16:16.117009 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:16.116968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:16.117018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117135 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:24.117119748 +0000 UTC m=+50.636387231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : configmap references non-existent config key: service-ca.crt Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117186 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117211 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86df76cfff-rzhm6: secret "image-registry-tls" not found Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117258 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:16.117186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117262 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls podName:3b2e70b6-bc6d-441d-8447-91ace8d34925 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:24.117245536 +0000 UTC m=+50.636513027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls") pod "image-registry-86df76cfff-rzhm6" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925") : secret "image-registry-tls" not found Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117316 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:24.117298078 +0000 UTC m=+50.636565558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : secret "router-metrics-certs-default" not found Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:16.117420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:16.117447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:16.117535 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:16.117503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:16.118009 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117567 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:16:16.118009 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117598 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:16:16.118009 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117612 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:16.118009 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117614 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls podName:81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:24.117602968 +0000 UTC m=+50.636870461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls") pod "dns-default-hm75f" (UID: "81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e") : secret "dns-default-metrics-tls" not found Sep 18 16:16:16.118009 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117655 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls podName:f1b73e02-8e1c-41db-a213-b8baee6c4e54 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:24.117645019 +0000 UTC m=+50.636912496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dxrt9" (UID: "f1b73e02-8e1c-41db-a213-b8baee6c4e54") : secret "samples-operator-tls" not found Sep 18 16:16:16.118009 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:16.117666 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls podName:9c9fda39-70e8-4d3b-9e16-d767847f72e0 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:24.117659479 +0000 UTC m=+50.636926956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4blfd" (UID: "9c9fda39-70e8-4d3b-9e16-d767847f72e0") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:17.874589 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:17.874563 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g" event={"ID":"52b086e1-8d64-4d18-93c6-f411c3cb7b87","Type":"ContainerStarted","Data":"267d1cfd8c8b3047755f5f443eac19a4a51b331157eabe81a6959171265518c7"} Sep 18 16:16:18.435857 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.435821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:18.436506 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:18.436465 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:16:18.436613 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:18.436575 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert podName:584c5c9c-2e21-4c1f-a309-67e12d7d592c nodeName:}" failed. No retries permitted until 2026-09-18 16:16:26.436550713 +0000 UTC m=+52.955818211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ff5dx" (UID: "584c5c9c-2e21-4c1f-a309-67e12d7d592c") : secret "networking-console-plugin-cert" not found Sep 18 16:16:18.880316 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.880240 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7xjxp" event={"ID":"02dcbb62-fa0e-491c-9428-59eb7856d6b2","Type":"ContainerStarted","Data":"997004fd8fc5e02bd9409cf7ce6863738ac771f9314c437e3be6dad218c38d58"} Sep 18 16:16:18.880699 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.880332 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:16:18.881702 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.881675 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv" event={"ID":"38551f86-17da-4945-84d1-b8f8cccbcd73","Type":"ContainerStarted","Data":"7dbf5fa16dec20bc1513e4a69768a2dadf082cd37ceee219a9e441195591dbce"} Sep 18 16:16:18.883203 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.883180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" event={"ID":"b0047f8a-de97-43ae-b9b5-679de1b69eef","Type":"ContainerStarted","Data":"596ec61388c2055dd361b016bd692414a808252d2f1706b3a0be047fe3a63f34"} Sep 18 16:16:18.884722 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.884703 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/0.log" Sep 18 16:16:18.884823 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.884736 2564 generic.go:356] "Generic (PLEG): container finished" podID="d14e4595-8b55-4f60-a557-aa750c699157" containerID="1ace46133802b990c9040bce1eb893b2e3557d16d14ed31521525e9ea46a6fc8" exitCode=255 Sep 18 16:16:18.884823 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.884793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" event={"ID":"d14e4595-8b55-4f60-a557-aa750c699157","Type":"ContainerDied","Data":"1ace46133802b990c9040bce1eb893b2e3557d16d14ed31521525e9ea46a6fc8"} Sep 18 16:16:18.884987 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.884970 2564 scope.go:117] "RemoveContainer" containerID="1ace46133802b990c9040bce1eb893b2e3557d16d14ed31521525e9ea46a6fc8" Sep 18 16:16:18.886396 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.886375 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-92smk" event={"ID":"87108184-50a2-4e70-9887-69a7b01af53f","Type":"ContainerStarted","Data":"a6c284df10e15b04375fdad1a7f5178552f921292969a1b7cca1e7eb14c56ded"} Sep 18 16:16:18.888050 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.888027 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" event={"ID":"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af","Type":"ContainerStarted","Data":"ecf02de3549ded48ff93a6d04d9e6fd2bef68f724683871e92b8382b1e8c92e9"} Sep 18 16:16:18.889738 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.889716 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" event={"ID":"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f","Type":"ContainerStarted","Data":"a209b52e08f780b7299147365960b7f60e3542395117298b25a73b20be061247"} Sep 18 16:16:18.960384 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.960332 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7xjxp" podStartSLOduration=36.946343807 podStartE2EDuration="43.960314271s" totalImagesPullingTime="7.013970464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:18.95521256 +0000 UTC m=+45.474480059" watchObservedRunningTime="2026-09-18 16:16:18.960314271 +0000 UTC m=+45.479581769" Sep 18 16:16:18.961159 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:18.961121 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-c5s8g" podStartSLOduration=13.723598463 podStartE2EDuration="16.961110286s" totalImagesPullingTime="3.237511823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:17.914174985 +0000 UTC m=+44.433442475" watchObservedRunningTime="2026-09-18 16:16:18.961110286 +0000 UTC m=+45.480377786" Sep 18 16:16:19.005515 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.005016 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-z55rv" podStartSLOduration=3.659660594 podStartE2EDuration="11.004996462s" totalImagesPullingTime="7.345335868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:19.002675078 +0000 UTC m=+45.521942578" watchObservedRunningTime="2026-09-18 16:16:19.004996462 +0000 UTC m=+45.524263963" Sep 18 16:16:19.025891 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.025821 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" podStartSLOduration=2.266992559 podStartE2EDuration="9.025804642s" totalImagesPullingTime="6.758812083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:19.025227154 +0000 UTC m=+45.544494658" watchObservedRunningTime="2026-09-18 16:16:19.025804642 +0000 UTC m=+45.545072142" Sep 18 16:16:19.033069 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.033044 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5zlr7"] Sep 18 16:16:19.053953 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.053929 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.054474 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.054416 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" podStartSLOduration=8.168977417 podStartE2EDuration="17.054399284s" totalImagesPullingTime="8.885421867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:19.051457997 +0000 UTC m=+45.570725496" watchObservedRunningTime="2026-09-18 16:16:19.054399284 +0000 UTC m=+45.573666785" Sep 18 16:16:19.055107 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.055091 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5zlr7"] Sep 18 16:16:19.057063 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.057036 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 18 16:16:19.057063 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.057054 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 18 16:16:19.057341 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.057324 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pj9mb\"" Sep 18 16:16:19.074477 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.073322 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" podStartSLOduration=8.219233494 podStartE2EDuration="17.073302499s" totalImagesPullingTime="8.854069005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:19.07256551 +0000 UTC m=+45.591833010" watchObservedRunningTime="2026-09-18 16:16:19.073302499 +0000 UTC m=+45.592569998" Sep 18 16:16:19.110084 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.110032 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-92smk" podStartSLOduration=2.284342305 podStartE2EDuration="11.110018733s" totalImagesPullingTime="8.825676428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:19.108962906 +0000 UTC m=+45.628230405" watchObservedRunningTime="2026-09-18 16:16:19.110018733 +0000 UTC m=+45.629286231" Sep 18 16:16:19.145173 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.145110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bckwp\" (UniqueName: \"kubernetes.io/projected/8326ef92-0807-48dd-a612-c5c8113c8591-kube-api-access-bckwp\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.145277 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.145194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.145277 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.145253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8326ef92-0807-48dd-a612-c5c8113c8591-crio-socket\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.145380 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.145277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.145380 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.145299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8326ef92-0807-48dd-a612-c5c8113c8591-data-volume\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.245953 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.245927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bckwp\" (UniqueName: \"kubernetes.io/projected/8326ef92-0807-48dd-a612-c5c8113c8591-kube-api-access-bckwp\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.246101 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.245987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.246101 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.246024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8326ef92-0807-48dd-a612-c5c8113c8591-crio-socket\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.246101 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.246046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.246259 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.246123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8326ef92-0807-48dd-a612-c5c8113c8591-crio-socket\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.246259 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:19.246138 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:16:19.246259 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:19.246176 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:16:19.246259 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:19.246194 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls podName:8326ef92-0807-48dd-a612-c5c8113c8591 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:19.746174965 +0000 UTC m=+46.265442456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zlr7" (UID: "8326ef92-0807-48dd-a612-c5c8113c8591") : secret "insights-runtime-extractor-tls" not found Sep 18 16:16:19.246259 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:19.246234 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm podName:8326ef92-0807-48dd-a612-c5c8113c8591 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:19.746220475 +0000 UTC m=+46.265487972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zlr7" (UID: "8326ef92-0807-48dd-a612-c5c8113c8591") : configmap "kube-rbac-proxy" not found Sep 18 16:16:19.246259 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.246249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8326ef92-0807-48dd-a612-c5c8113c8591-data-volume\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.246565 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.246549 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8326ef92-0807-48dd-a612-c5c8113c8591-data-volume\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.255087 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.255058 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bckwp\" (UniqueName: \"kubernetes.io/projected/8326ef92-0807-48dd-a612-c5c8113c8591-kube-api-access-bckwp\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.750938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.750903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.751137 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.750947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:19.751137 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:19.751050 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:16:19.751137 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:19.751080 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:16:19.751137 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:19.751120 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm podName:8326ef92-0807-48dd-a612-c5c8113c8591 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:20.751101493 +0000 UTC m=+47.270368981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zlr7" (UID: "8326ef92-0807-48dd-a612-c5c8113c8591") : configmap "kube-rbac-proxy" not found Sep 18 16:16:19.751311 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:19.751139 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls podName:8326ef92-0807-48dd-a612-c5c8113c8591 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:20.751129893 +0000 UTC m=+47.270397370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zlr7" (UID: "8326ef92-0807-48dd-a612-c5c8113c8591") : secret "insights-runtime-extractor-tls" not found Sep 18 16:16:19.897346 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.897312 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:16:19.897787 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.897768 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/0.log" Sep 18 16:16:19.897852 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.897808 2564 generic.go:356] "Generic (PLEG): container finished" podID="d14e4595-8b55-4f60-a557-aa750c699157" containerID="cdeb1032170520dc9b174ac9039820267fee5e837f513c195e0bfe37a9b2f01b" exitCode=255 Sep 18 16:16:19.897917 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.897891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" event={"ID":"d14e4595-8b55-4f60-a557-aa750c699157","Type":"ContainerDied","Data":"cdeb1032170520dc9b174ac9039820267fee5e837f513c195e0bfe37a9b2f01b"} Sep 18 16:16:19.897995 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.897980 2564 scope.go:117] "RemoveContainer" containerID="1ace46133802b990c9040bce1eb893b2e3557d16d14ed31521525e9ea46a6fc8" Sep 18 16:16:19.898119 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:19.898095 2564 scope.go:117] "RemoveContainer" containerID="cdeb1032170520dc9b174ac9039820267fee5e837f513c195e0bfe37a9b2f01b" Sep 18 16:16:19.898331 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:19.898299 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-mbpfm_openshift-console-operator(d14e4595-8b55-4f60-a557-aa750c699157)\"" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" podUID="d14e4595-8b55-4f60-a557-aa750c699157" Sep 18 16:16:20.352307 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:20.352280 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8vqrx_992c20b0-878a-4325-82d7-10fa9f9acd06/dns-node-resolver/0.log" Sep 18 16:16:20.761003 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:20.760966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:20.761165 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:20.761015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:20.761165 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:20.761087 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:16:20.761165 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:20.761104 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:16:20.761165 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:20.761151 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm podName:8326ef92-0807-48dd-a612-c5c8113c8591 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:22.761136118 +0000 UTC m=+49.280403600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zlr7" (UID: "8326ef92-0807-48dd-a612-c5c8113c8591") : configmap "kube-rbac-proxy" not found Sep 18 16:16:20.761290 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:20.761167 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls podName:8326ef92-0807-48dd-a612-c5c8113c8591 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:22.761159809 +0000 UTC m=+49.280427286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zlr7" (UID: "8326ef92-0807-48dd-a612-c5c8113c8591") : secret "insights-runtime-extractor-tls" not found Sep 18 16:16:20.902341 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:20.902320 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:16:20.902690 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:20.902646 2564 scope.go:117] "RemoveContainer" containerID="cdeb1032170520dc9b174ac9039820267fee5e837f513c195e0bfe37a9b2f01b" Sep 18 16:16:20.902805 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:20.902786 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-mbpfm_openshift-console-operator(d14e4595-8b55-4f60-a557-aa750c699157)\"" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" podUID="d14e4595-8b55-4f60-a557-aa750c699157" Sep 18 16:16:21.553154 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:21.553131 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wztrg_5e1fc1cd-109e-4bb8-aa37-19c71590b563/node-ca/0.log" Sep 18 16:16:22.153332 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:22.153306 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-92smk_87108184-50a2-4e70-9887-69a7b01af53f/serve-healthcheck-canary/0.log" Sep 18 16:16:22.776681 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:22.776640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:22.776846 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:22.776691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:22.776846 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:22.776770 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:16:22.776846 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:22.776807 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:16:22.776970 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:22.776851 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm podName:8326ef92-0807-48dd-a612-c5c8113c8591 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:26.776833578 +0000 UTC m=+53.296101076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zlr7" (UID: "8326ef92-0807-48dd-a612-c5c8113c8591") : configmap "kube-rbac-proxy" not found Sep 18 16:16:22.776970 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:22.776885 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls podName:8326ef92-0807-48dd-a612-c5c8113c8591 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:26.776872916 +0000 UTC m=+53.296140407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zlr7" (UID: "8326ef92-0807-48dd-a612-c5c8113c8591") : secret "insights-runtime-extractor-tls" not found Sep 18 16:16:24.188205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:24.188173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:24.188205 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:24.188205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:24.188231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:24.188268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.188364 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.188384 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.188398 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:40.188381649 +0000 UTC m=+66.707649162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : configmap references non-existent config key: service-ca.crt Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:24.188381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.188436 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls podName:81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:40.188418254 +0000 UTC m=+66.707685772 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls") pod "dns-default-hm75f" (UID: "81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e") : secret "dns-default-metrics-tls" not found Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.188473 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.188461 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls podName:9c9fda39-70e8-4d3b-9e16-d767847f72e0 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:40.188446866 +0000 UTC m=+66.707714344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4blfd" (UID: "9c9fda39-70e8-4d3b-9e16-d767847f72e0") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.188515 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86df76cfff-rzhm6: secret "image-registry-tls" not found Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.188553 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.188581 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls podName:3b2e70b6-bc6d-441d-8447-91ace8d34925 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:40.188565159 +0000 UTC m=+66.707832670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls") pod "image-registry-86df76cfff-rzhm6" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925") : secret "image-registry-tls" not found Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.188607 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:16:24.188768 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:24.188561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:24.189166 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.189080 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls podName:f1b73e02-8e1c-41db-a213-b8baee6c4e54 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:40.18859015 +0000 UTC m=+66.707857627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dxrt9" (UID: "f1b73e02-8e1c-41db-a213-b8baee6c4e54") : secret "samples-operator-tls" not found Sep 18 16:16:24.189166 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:24.189106 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs podName:73610849-9bcc-4084-a60e-a0ee6a2a7861 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:40.18909888 +0000 UTC m=+66.708366357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs") pod "router-default-787796f9d4-z4dch" (UID: "73610849-9bcc-4084-a60e-a0ee6a2a7861") : secret "router-metrics-certs-default" not found Sep 18 16:16:26.509961 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:26.509921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:26.510475 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:26.510111 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:16:26.510475 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:26.510221 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert podName:584c5c9c-2e21-4c1f-a309-67e12d7d592c nodeName:}" failed. No retries permitted until 2026-09-18 16:16:42.510196513 +0000 UTC m=+69.029464001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ff5dx" (UID: "584c5c9c-2e21-4c1f-a309-67e12d7d592c") : secret "networking-console-plugin-cert" not found Sep 18 16:16:26.813228 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:26.813139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:26.813228 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:26.813191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:26.813524 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:26.813303 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:16:26.813524 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:26.813377 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls podName:8326ef92-0807-48dd-a612-c5c8113c8591 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:34.813356635 +0000 UTC m=+61.332624162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zlr7" (UID: "8326ef92-0807-48dd-a612-c5c8113c8591") : secret "insights-runtime-extractor-tls" not found Sep 18 16:16:26.813809 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:26.813785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8326ef92-0807-48dd-a612-c5c8113c8591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:28.628096 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:28.628039 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:28.628516 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:28.628133 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:28.628516 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:28.628464 2564 scope.go:117] "RemoveContainer" containerID="cdeb1032170520dc9b174ac9039820267fee5e837f513c195e0bfe37a9b2f01b" Sep 18 16:16:28.628678 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:16:28.628653 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-mbpfm_openshift-console-operator(d14e4595-8b55-4f60-a557-aa750c699157)\"" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" podUID="d14e4595-8b55-4f60-a557-aa750c699157" Sep 18 16:16:33.804231 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:33.804198 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w9n9q" Sep 18 16:16:34.879458 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:34.879429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:34.881578 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:34.881555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8326ef92-0807-48dd-a612-c5c8113c8591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zlr7\" (UID: \"8326ef92-0807-48dd-a612-c5c8113c8591\") " pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:34.967476 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:34.967447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5zlr7" Sep 18 16:16:35.240540 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:35.240505 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5zlr7"] Sep 18 16:16:35.245114 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:35.245088 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8326ef92_0807_48dd_a612_c5c8113c8591.slice/crio-61fa3e2d48419f723f9f4d6c0d691d071a4334ea055b3dc9843b8e39e9f6d9de WatchSource:0}: Error finding container 61fa3e2d48419f723f9f4d6c0d691d071a4334ea055b3dc9843b8e39e9f6d9de: Status 404 returned error can't find the container with id 61fa3e2d48419f723f9f4d6c0d691d071a4334ea055b3dc9843b8e39e9f6d9de Sep 18 16:16:35.950629 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:35.950595 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zlr7" event={"ID":"8326ef92-0807-48dd-a612-c5c8113c8591","Type":"ContainerStarted","Data":"1925f4c587b37d3e8f36b3f09cf0865af69013297b8c691ce01b6cf1956d5f39"} Sep 18 16:16:35.950969 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:35.950651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zlr7" event={"ID":"8326ef92-0807-48dd-a612-c5c8113c8591","Type":"ContainerStarted","Data":"61fa3e2d48419f723f9f4d6c0d691d071a4334ea055b3dc9843b8e39e9f6d9de"} Sep 18 16:16:36.955763 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:36.955731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zlr7" event={"ID":"8326ef92-0807-48dd-a612-c5c8113c8591","Type":"ContainerStarted","Data":"a2b9d0a2ccc70955ea0e1aef2d50f2e0be9ee9fe0f3e1402cb8f1e4f0344d2bc"} Sep 18 16:16:37.962049 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:37.962007 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zlr7" event={"ID":"8326ef92-0807-48dd-a612-c5c8113c8591","Type":"ContainerStarted","Data":"99a85e1340e26e36d68a81e188507f531e95b7a66fe312fd3e37db7d091ab970"} Sep 18 16:16:37.981367 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:37.981304 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5zlr7" podStartSLOduration=16.764752586 podStartE2EDuration="18.981288262s" totalImagesPullingTime="2.216535676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:37.979902456 +0000 UTC m=+64.499169955" watchObservedRunningTime="2026-09-18 16:16:37.981288262 +0000 UTC m=+64.500555760" Sep 18 16:16:40.224140 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.224089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:40.224140 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.224136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:40.224140 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.224158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:40.224778 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.224196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:40.224778 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.224216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:40.224778 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.224238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:40.225141 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.225106 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73610849-9bcc-4084-a60e-a0ee6a2a7861-service-ca-bundle\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:40.227181 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.227141 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73610849-9bcc-4084-a60e-a0ee6a2a7861-metrics-certs\") pod \"router-default-787796f9d4-z4dch\" (UID: \"73610849-9bcc-4084-a60e-a0ee6a2a7861\") " pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:40.227513 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.227462 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e-metrics-tls\") pod \"dns-default-hm75f\" (UID: \"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e\") " pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:40.227625 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.227523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1b73e02-8e1c-41db-a213-b8baee6c4e54-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dxrt9\" (UID: \"f1b73e02-8e1c-41db-a213-b8baee6c4e54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:40.227687 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.227657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c9fda39-70e8-4d3b-9e16-d767847f72e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4blfd\" (UID: \"9c9fda39-70e8-4d3b-9e16-d767847f72e0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:40.227832 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.227810 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") pod \"image-registry-86df76cfff-rzhm6\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:40.390240 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.390215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h482k\"" Sep 18 16:16:40.397280 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.397258 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:40.412403 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.412382 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cxft6\"" Sep 18 16:16:40.420570 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.420542 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" Sep 18 16:16:40.434913 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.434891 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rjpnh\"" Sep 18 16:16:40.443172 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.443102 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:40.462984 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.450206 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5rtvs\"" Sep 18 16:16:40.469505 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.463360 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" Sep 18 16:16:40.484761 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.479779 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-44fnz\"" Sep 18 16:16:40.484761 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.480031 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:40.697803 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.697777 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86df76cfff-rzhm6"] Sep 18 16:16:40.700854 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:40.700831 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2e70b6_bc6d_441d_8447_91ace8d34925.slice/crio-6a95297c764fc23599bdc189b525f30fb75816f3590bcedbd596128f826f45f0 WatchSource:0}: Error finding container 6a95297c764fc23599bdc189b525f30fb75816f3590bcedbd596128f826f45f0: Status 404 returned error can't find the container with id 6a95297c764fc23599bdc189b525f30fb75816f3590bcedbd596128f826f45f0 Sep 18 16:16:40.718406 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.718384 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd"] Sep 18 16:16:40.720467 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:40.720442 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9fda39_70e8_4d3b_9e16_d767847f72e0.slice/crio-b089b971596553c413502e63993a10ad13a1c090f5211a4df4591e8b9ae14be7 WatchSource:0}: Error finding container b089b971596553c413502e63993a10ad13a1c090f5211a4df4591e8b9ae14be7: Status 404 returned error can't find the container with id b089b971596553c413502e63993a10ad13a1c090f5211a4df4591e8b9ae14be7 Sep 18 16:16:40.764003 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.763968 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-787796f9d4-z4dch"] Sep 18 16:16:40.766686 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:40.766656 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73610849_9bcc_4084_a60e_a0ee6a2a7861.slice/crio-c5850e1e20150acfa9feceb18dbb2aada47f10890d95ab3f280c6d3e78610442 WatchSource:0}: Error finding container c5850e1e20150acfa9feceb18dbb2aada47f10890d95ab3f280c6d3e78610442: Status 404 returned error can't find the container with id c5850e1e20150acfa9feceb18dbb2aada47f10890d95ab3f280c6d3e78610442 Sep 18 16:16:40.785054 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.785032 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9"] Sep 18 16:16:40.786458 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.786350 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hm75f"] Sep 18 16:16:40.787006 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:40.786986 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e2c2c3_c7ae_4ded_980c_c23c9cc4eb1e.slice/crio-862ec42e7e9546055881f81f3448bc8a735758b18f70a5c6a1afb8114d4759c9 WatchSource:0}: Error finding container 862ec42e7e9546055881f81f3448bc8a735758b18f70a5c6a1afb8114d4759c9: Status 404 returned error can't find the container with id 862ec42e7e9546055881f81f3448bc8a735758b18f70a5c6a1afb8114d4759c9 Sep 18 16:16:40.974679 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.974640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hm75f" event={"ID":"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e","Type":"ContainerStarted","Data":"862ec42e7e9546055881f81f3448bc8a735758b18f70a5c6a1afb8114d4759c9"} Sep 18 16:16:40.980456 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.980422 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-787796f9d4-z4dch" event={"ID":"73610849-9bcc-4084-a60e-a0ee6a2a7861","Type":"ContainerStarted","Data":"2a468b4cf69c5c98711d4bfe50973fea7b02d6a6455abdd38f14d5261b862259"} Sep 18 16:16:40.980605 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.980533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-787796f9d4-z4dch" event={"ID":"73610849-9bcc-4084-a60e-a0ee6a2a7861","Type":"ContainerStarted","Data":"c5850e1e20150acfa9feceb18dbb2aada47f10890d95ab3f280c6d3e78610442"} Sep 18 16:16:40.981756 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.981730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" event={"ID":"9c9fda39-70e8-4d3b-9e16-d767847f72e0","Type":"ContainerStarted","Data":"b089b971596553c413502e63993a10ad13a1c090f5211a4df4591e8b9ae14be7"} Sep 18 16:16:40.983398 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.983366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" event={"ID":"3b2e70b6-bc6d-441d-8447-91ace8d34925","Type":"ContainerStarted","Data":"a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2"} Sep 18 16:16:40.983398 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.983398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" event={"ID":"3b2e70b6-bc6d-441d-8447-91ace8d34925","Type":"ContainerStarted","Data":"6a95297c764fc23599bdc189b525f30fb75816f3590bcedbd596128f826f45f0"} Sep 18 16:16:40.983570 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.983415 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:16:40.984643 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.984620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" event={"ID":"f1b73e02-8e1c-41db-a213-b8baee6c4e54","Type":"ContainerStarted","Data":"aef9119edc34f68d497b0fb019f41357f9cf348578cafaae2bf55cdb84f45e0c"} Sep 18 16:16:40.999937 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:40.999856 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-787796f9d4-z4dch" podStartSLOduration=38.999844604 podStartE2EDuration="38.999844604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:40.998863654 +0000 UTC m=+67.518131154" watchObservedRunningTime="2026-09-18 16:16:40.999844604 +0000 UTC m=+67.519112102" Sep 18 16:16:41.018058 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.017785 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" podStartSLOduration=65.017768394 podStartE2EDuration="1m5.017768394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:41.016724365 +0000 UTC m=+67.535991877" watchObservedRunningTime="2026-09-18 16:16:41.017768394 +0000 UTC m=+67.537035894" Sep 18 16:16:41.333309 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.333226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:16:41.336525 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.336504 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 16:16:41.345816 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.345789 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415ad465-e1a1-4c60-99dd-7c18748d64e8-metrics-certs\") pod \"network-metrics-daemon-r4pj7\" (UID: \"415ad465-e1a1-4c60-99dd-7c18748d64e8\") " pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:16:41.434108 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.434055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:16:41.437048 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.437022 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 16:16:41.443503 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.443454 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:41.446379 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.446348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:41.447616 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.447592 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/458d51a9-e012-413e-8dd6-8354a56b8abc-original-pull-secret\") pod \"global-pull-secret-syncer-msncf\" (UID: \"458d51a9-e012-413e-8dd6-8354a56b8abc\") " pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:16:41.448865 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.448843 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kmlw\"" Sep 18 16:16:41.451952 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.451905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msncf" Sep 18 16:16:41.457299 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.457279 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4pj7" Sep 18 16:16:41.751726 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.751663 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-msncf"] Sep 18 16:16:41.755153 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:41.755125 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458d51a9_e012_413e_8dd6_8354a56b8abc.slice/crio-58608b7da29144fc8122f3293deee4fd962748b77e86c876e1dd02a5e8bb292b WatchSource:0}: Error finding container 58608b7da29144fc8122f3293deee4fd962748b77e86c876e1dd02a5e8bb292b: Status 404 returned error can't find the container with id 58608b7da29144fc8122f3293deee4fd962748b77e86c876e1dd02a5e8bb292b Sep 18 16:16:41.782757 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.782683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r4pj7"] Sep 18 16:16:41.787084 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:41.787055 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415ad465_e1a1_4c60_99dd_7c18748d64e8.slice/crio-f1dcd1f34852402448a7ffa7cfc4bbd1513d40a64e3d8e2ad3cade7bff77860d WatchSource:0}: Error finding container f1dcd1f34852402448a7ffa7cfc4bbd1513d40a64e3d8e2ad3cade7bff77860d: Status 404 returned error can't find the container with id f1dcd1f34852402448a7ffa7cfc4bbd1513d40a64e3d8e2ad3cade7bff77860d Sep 18 16:16:41.990612 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.990571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4pj7" event={"ID":"415ad465-e1a1-4c60-99dd-7c18748d64e8","Type":"ContainerStarted","Data":"f1dcd1f34852402448a7ffa7cfc4bbd1513d40a64e3d8e2ad3cade7bff77860d"} Sep 18 16:16:41.992040 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.991970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-msncf" event={"ID":"458d51a9-e012-413e-8dd6-8354a56b8abc","Type":"ContainerStarted","Data":"58608b7da29144fc8122f3293deee4fd962748b77e86c876e1dd02a5e8bb292b"} Sep 18 16:16:41.992463 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.992440 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:41.993848 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:41.993826 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-787796f9d4-z4dch" Sep 18 16:16:42.547728 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:42.547690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:42.553159 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:42.553129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/584c5c9c-2e21-4c1f-a309-67e12d7d592c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ff5dx\" (UID: \"584c5c9c-2e21-4c1f-a309-67e12d7d592c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:42.780867 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:42.780765 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-m9mjg\"" Sep 18 16:16:42.788935 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:42.788909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" Sep 18 16:16:43.635384 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:43.635355 2564 scope.go:117] "RemoveContainer" containerID="cdeb1032170520dc9b174ac9039820267fee5e837f513c195e0bfe37a9b2f01b" Sep 18 16:16:46.776815 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:46.776693 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx"] Sep 18 16:16:46.786146 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:46.786118 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584c5c9c_2e21_4c1f_a309_67e12d7d592c.slice/crio-3dae60d2a7933e7946cf6ceaa3a827d9384684b562f045ca775e1f0cc67f72d1 WatchSource:0}: Error finding container 3dae60d2a7933e7946cf6ceaa3a827d9384684b562f045ca775e1f0cc67f72d1: Status 404 returned error can't find the container with id 3dae60d2a7933e7946cf6ceaa3a827d9384684b562f045ca775e1f0cc67f72d1 Sep 18 16:16:47.016787 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.016232 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hm75f" event={"ID":"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e","Type":"ContainerStarted","Data":"717323a132977795103da68052267796e5ea85594db86581bc0715ec0123ff91"} Sep 18 16:16:47.016787 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.016291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hm75f" event={"ID":"81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e","Type":"ContainerStarted","Data":"dc5bc6db49dd6c00e465a01bf8776d7a250268998096da6daa489d40817e9d48"} Sep 18 16:16:47.016787 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.016501 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:47.018293 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.018270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" event={"ID":"9c9fda39-70e8-4d3b-9e16-d767847f72e0","Type":"ContainerStarted","Data":"a746ef419f4aab1d731fea311b049bcb91568ddd796c5dae91cd121a20f35550"} Sep 18 16:16:47.020243 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.020205 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:16:47.020352 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.020295 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" event={"ID":"d14e4595-8b55-4f60-a557-aa750c699157","Type":"ContainerStarted","Data":"539e9b7d9a969d3a30b139098ebbdf7d9c26471cfafefe34fd7d3d78286340bf"} Sep 18 16:16:47.020700 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.020655 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:47.022831 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.022770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4pj7" event={"ID":"415ad465-e1a1-4c60-99dd-7c18748d64e8","Type":"ContainerStarted","Data":"a035407674fa4abbadb48544efa274c781e70c08c582472813d2cf4e965a8796"} Sep 18 16:16:47.022925 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.022851 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4pj7" event={"ID":"415ad465-e1a1-4c60-99dd-7c18748d64e8","Type":"ContainerStarted","Data":"5f93f4010bf4befb51ba39cfc062315699869e8f8b4a5ebc3de213f8ace9f40f"} Sep 18 16:16:47.024167 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.024127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" event={"ID":"584c5c9c-2e21-4c1f-a309-67e12d7d592c","Type":"ContainerStarted","Data":"3dae60d2a7933e7946cf6ceaa3a827d9384684b562f045ca775e1f0cc67f72d1"} Sep 18 16:16:47.026288 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.026270 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" Sep 18 16:16:47.027512 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.027150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" event={"ID":"f1b73e02-8e1c-41db-a213-b8baee6c4e54","Type":"ContainerStarted","Data":"9be2557e347834cd5dfee77e0b209062bfe97b7f495c1b03642cb0c072222d6b"} Sep 18 16:16:47.027512 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.027192 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" event={"ID":"f1b73e02-8e1c-41db-a213-b8baee6c4e54","Type":"ContainerStarted","Data":"f320ac6eeaf520b1b6ff9a5b59d2f77007cb21bed4d624dfc4c0e018f37fb56b"} Sep 18 16:16:47.035880 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.035812 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hm75f" podStartSLOduration=33.311233237 podStartE2EDuration="39.035793639s" totalImagesPullingTime="5.724560402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:47.03475352 +0000 UTC m=+73.554021019" watchObservedRunningTime="2026-09-18 16:16:47.035793639 +0000 UTC m=+73.555061138" Sep 18 16:16:47.061772 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.061720 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-msncf" podStartSLOduration=66.005176518 podStartE2EDuration="1m11.061706328s" totalImagesPullingTime="5.05652981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:47.060998248 +0000 UTC m=+73.580265745" watchObservedRunningTime="2026-09-18 16:16:47.061706328 +0000 UTC m=+73.580973818" Sep 18 16:16:47.100022 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.099964 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4blfd" podStartSLOduration=39.316210476 podStartE2EDuration="45.099943532s" totalImagesPullingTime="5.783733056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:47.097883612 +0000 UTC m=+73.617151113" watchObservedRunningTime="2026-09-18 16:16:47.099943532 +0000 UTC m=+73.619211032" Sep 18 16:16:47.141071 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.141004 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-mbpfm" podStartSLOduration=35.29100755 podStartE2EDuration="44.140987932s" totalImagesPullingTime="8.849980382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:47.139399778 +0000 UTC m=+73.658667277" watchObservedRunningTime="2026-09-18 16:16:47.140987932 +0000 UTC m=+73.660255430" Sep 18 16:16:47.184705 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.184651 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dxrt9" podStartSLOduration=39.494322713 podStartE2EDuration="45.18462948s" totalImagesPullingTime="5.690306767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:47.18328755 +0000 UTC m=+73.702555049" watchObservedRunningTime="2026-09-18 16:16:47.18462948 +0000 UTC m=+73.703896979" Sep 18 16:16:47.219644 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:47.219471 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r4pj7" podStartSLOduration=67.498521538 podStartE2EDuration="1m12.219449084s" totalImagesPullingTime="4.720927546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:47.219024769 +0000 UTC m=+73.738292311" watchObservedRunningTime="2026-09-18 16:16:47.219449084 +0000 UTC m=+73.738716605" Sep 18 16:16:48.034294 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:48.034255 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-msncf" event={"ID":"458d51a9-e012-413e-8dd6-8354a56b8abc","Type":"ContainerStarted","Data":"f8529ab0492b741490805a8751c4dd5551a661de14f4b860bf2ffc2e1e5fecd2"} Sep 18 16:16:49.039933 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:49.039899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" event={"ID":"584c5c9c-2e21-4c1f-a309-67e12d7d592c","Type":"ContainerStarted","Data":"b3541bdc92a9e494515daebc84fd2aaf377ece181600b0caf7765b992e729fc7"} Sep 18 16:16:49.057150 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:49.057103 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ff5dx" podStartSLOduration=37.601732547 podStartE2EDuration="39.05708825s" totalImagesPullingTime="1.455355703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:49.056852257 +0000 UTC m=+75.576119757" watchObservedRunningTime="2026-09-18 16:16:49.05708825 +0000 UTC m=+75.576355749" Sep 18 16:16:49.901296 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:49.901261 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7xjxp" Sep 18 16:16:57.037039 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.037009 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hm75f" Sep 18 16:16:57.779254 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.779221 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4f8tj"] Sep 18 16:16:57.781587 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.781566 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.784274 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.784250 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 18 16:16:57.784370 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.784262 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 18 16:16:57.785319 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.785299 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 18 16:16:57.785319 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.785297 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 18 16:16:57.785598 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.785584 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vpzqb\"" Sep 18 16:16:57.857497 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.857469 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e6305e8-22de-493f-a8f7-b02a6b32e066-metrics-client-ca\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.857598 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.857514 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-wtmp\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.857598 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.857533 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6305e8-22de-493f-a8f7-b02a6b32e066-sys\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.857598 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.857549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-textfile\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.857746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.857645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e6305e8-22de-493f-a8f7-b02a6b32e066-root\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.857746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.857703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-tls\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.857825 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.857737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.857825 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.857772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7rqq\" (UniqueName: \"kubernetes.io/projected/6e6305e8-22de-493f-a8f7-b02a6b32e066-kube-api-access-z7rqq\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.857825 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.857813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-accelerators-collector-config\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959141 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e6305e8-22de-493f-a8f7-b02a6b32e066-root\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959285 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-tls\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959285 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959285 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7rqq\" (UniqueName: \"kubernetes.io/projected/6e6305e8-22de-493f-a8f7-b02a6b32e066-kube-api-access-z7rqq\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959285 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959222 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e6305e8-22de-493f-a8f7-b02a6b32e066-root\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959285 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-accelerators-collector-config\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959285 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e6305e8-22de-493f-a8f7-b02a6b32e066-metrics-client-ca\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959595 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-wtmp\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959595 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6305e8-22de-493f-a8f7-b02a6b32e066-sys\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959595 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959365 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-textfile\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.959749 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-textfile\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.960323 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959876 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-wtmp\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.960323 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.959936 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e6305e8-22de-493f-a8f7-b02a6b32e066-sys\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.960323 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.960063 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e6305e8-22de-493f-a8f7-b02a6b32e066-metrics-client-ca\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.960323 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.960129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-accelerators-collector-config\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.961938 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.961909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-tls\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.962237 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.962210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e6305e8-22de-493f-a8f7-b02a6b32e066-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:57.967145 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:57.967120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7rqq\" (UniqueName: \"kubernetes.io/projected/6e6305e8-22de-493f-a8f7-b02a6b32e066-kube-api-access-z7rqq\") pod \"node-exporter-4f8tj\" (UID: \"6e6305e8-22de-493f-a8f7-b02a6b32e066\") " pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:58.090417 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:58.090369 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4f8tj" Sep 18 16:16:58.102039 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:16:58.102017 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6305e8_22de_493f_a8f7_b02a6b32e066.slice/crio-20663581ff5556ada7caa5b8634510929db119db458a24f2f7436b549fc0ece9 WatchSource:0}: Error finding container 20663581ff5556ada7caa5b8634510929db119db458a24f2f7436b549fc0ece9: Status 404 returned error can't find the container with id 20663581ff5556ada7caa5b8634510929db119db458a24f2f7436b549fc0ece9 Sep 18 16:16:59.078838 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:16:59.078801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4f8tj" event={"ID":"6e6305e8-22de-493f-a8f7-b02a6b32e066","Type":"ContainerStarted","Data":"20663581ff5556ada7caa5b8634510929db119db458a24f2f7436b549fc0ece9"} Sep 18 16:17:00.084286 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:00.084248 2564 generic.go:356] "Generic (PLEG): container finished" podID="6e6305e8-22de-493f-a8f7-b02a6b32e066" containerID="96be9f527ff6fa505bfed3870660b3aef5eddd018827ce7dbb3939e928ff9412" exitCode=0 Sep 18 16:17:00.084733 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:00.084336 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4f8tj" event={"ID":"6e6305e8-22de-493f-a8f7-b02a6b32e066","Type":"ContainerDied","Data":"96be9f527ff6fa505bfed3870660b3aef5eddd018827ce7dbb3939e928ff9412"} Sep 18 16:17:01.091291 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:01.091253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4f8tj" event={"ID":"6e6305e8-22de-493f-a8f7-b02a6b32e066","Type":"ContainerStarted","Data":"649dfa5ecc11efd1a064f61c29e5a186672eb9f752143b803428419ecb6c11be"} Sep 18 16:17:01.091690 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:01.091310 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4f8tj" event={"ID":"6e6305e8-22de-493f-a8f7-b02a6b32e066","Type":"ContainerStarted","Data":"1c3015f942f0a0fc6fd67cbb7eadef78fe45c211e62784285702d94ec87cf855"} Sep 18 16:17:01.119569 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:01.119519 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4f8tj" podStartSLOduration=2.934921564 podStartE2EDuration="4.119481123s" totalImagesPullingTime="1.125010357s" totalInitContainerRuntime="59.549202ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:01.118453107 +0000 UTC m=+87.637720651" watchObservedRunningTime="2026-09-18 16:17:01.119481123 +0000 UTC m=+87.638748621" Sep 18 16:17:01.997171 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:01.997141 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:17:08.390522 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:08.390468 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86df76cfff-rzhm6"] Sep 18 16:17:24.178527 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:24.178468 2564 generic.go:356] "Generic (PLEG): container finished" podID="b0047f8a-de97-43ae-b9b5-679de1b69eef" containerID="596ec61388c2055dd361b016bd692414a808252d2f1706b3a0be047fe3a63f34" exitCode=0 Sep 18 16:17:24.178930 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:24.178545 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" event={"ID":"b0047f8a-de97-43ae-b9b5-679de1b69eef","Type":"ContainerDied","Data":"596ec61388c2055dd361b016bd692414a808252d2f1706b3a0be047fe3a63f34"} Sep 18 16:17:24.178930 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:24.178877 2564 scope.go:117] "RemoveContainer" containerID="596ec61388c2055dd361b016bd692414a808252d2f1706b3a0be047fe3a63f34" Sep 18 16:17:25.184771 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:25.184734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-6zxcq" event={"ID":"b0047f8a-de97-43ae-b9b5-679de1b69eef","Type":"ContainerStarted","Data":"7369e8a74a4cc369b517f4c4d962730532256e3b80ff7305f5ecccc960a02cdb"} Sep 18 16:17:33.409325 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.409285 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" podUID="3b2e70b6-bc6d-441d-8447-91ace8d34925" containerName="registry" containerID="cri-o://a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2" gracePeriod=30 Sep 18 16:17:33.642539 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.642517 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:17:33.795500 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.795463 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-image-registry-private-configuration\") pod \"3b2e70b6-bc6d-441d-8447-91ace8d34925\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " Sep 18 16:17:33.795628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.795517 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b2e70b6-bc6d-441d-8447-91ace8d34925-ca-trust-extracted\") pod \"3b2e70b6-bc6d-441d-8447-91ace8d34925\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " Sep 18 16:17:33.795628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.795535 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-bound-sa-token\") pod \"3b2e70b6-bc6d-441d-8447-91ace8d34925\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " Sep 18 16:17:33.795628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.795578 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") pod \"3b2e70b6-bc6d-441d-8447-91ace8d34925\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " Sep 18 16:17:33.795628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.795602 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-installation-pull-secrets\") pod \"3b2e70b6-bc6d-441d-8447-91ace8d34925\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " Sep 18 16:17:33.795837 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.795630 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-certificates\") pod \"3b2e70b6-bc6d-441d-8447-91ace8d34925\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " Sep 18 16:17:33.795837 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.795653 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6ghs\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-kube-api-access-m6ghs\") pod \"3b2e70b6-bc6d-441d-8447-91ace8d34925\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " Sep 18 16:17:33.795837 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.795680 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-trusted-ca\") pod \"3b2e70b6-bc6d-441d-8447-91ace8d34925\" (UID: \"3b2e70b6-bc6d-441d-8447-91ace8d34925\") " Sep 18 16:17:33.796325 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.796251 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3b2e70b6-bc6d-441d-8447-91ace8d34925" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:17:33.796822 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.796777 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3b2e70b6-bc6d-441d-8447-91ace8d34925" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:17:33.798434 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.798344 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3b2e70b6-bc6d-441d-8447-91ace8d34925" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:17:33.798434 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.798409 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3b2e70b6-bc6d-441d-8447-91ace8d34925" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:17:33.798574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.798436 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3b2e70b6-bc6d-441d-8447-91ace8d34925" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:17:33.798574 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.798476 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3b2e70b6-bc6d-441d-8447-91ace8d34925" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:17:33.799057 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.799030 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-kube-api-access-m6ghs" (OuterVolumeSpecName: "kube-api-access-m6ghs") pod "3b2e70b6-bc6d-441d-8447-91ace8d34925" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925"). InnerVolumeSpecName "kube-api-access-m6ghs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:17:33.804072 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.804049 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2e70b6-bc6d-441d-8447-91ace8d34925-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3b2e70b6-bc6d-441d-8447-91ace8d34925" (UID: "3b2e70b6-bc6d-441d-8447-91ace8d34925"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:17:33.896846 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.896793 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-bound-sa-token\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:17:33.896846 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.896815 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:17:33.896846 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.896824 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-installation-pull-secrets\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:17:33.896846 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.896834 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-registry-certificates\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:17:33.896846 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.896842 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6ghs\" (UniqueName: \"kubernetes.io/projected/3b2e70b6-bc6d-441d-8447-91ace8d34925-kube-api-access-m6ghs\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:17:33.897037 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.896850 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b2e70b6-bc6d-441d-8447-91ace8d34925-trusted-ca\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:17:33.897037 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.896860 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b2e70b6-bc6d-441d-8447-91ace8d34925-image-registry-private-configuration\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:17:33.897037 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:33.896869 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b2e70b6-bc6d-441d-8447-91ace8d34925-ca-trust-extracted\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:17:34.220043 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.220011 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f" containerID="a209b52e08f780b7299147365960b7f60e3542395117298b25a73b20be061247" exitCode=0 Sep 18 16:17:34.220152 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.220090 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" event={"ID":"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f","Type":"ContainerDied","Data":"a209b52e08f780b7299147365960b7f60e3542395117298b25a73b20be061247"} Sep 18 16:17:34.220478 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.220460 2564 scope.go:117] "RemoveContainer" containerID="a209b52e08f780b7299147365960b7f60e3542395117298b25a73b20be061247" Sep 18 16:17:34.221700 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.221672 2564 generic.go:356] "Generic (PLEG): container finished" podID="3b2e70b6-bc6d-441d-8447-91ace8d34925" containerID="a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2" exitCode=0 Sep 18 16:17:34.221820 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.221738 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" Sep 18 16:17:34.221820 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.221751 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" event={"ID":"3b2e70b6-bc6d-441d-8447-91ace8d34925","Type":"ContainerDied","Data":"a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2"} Sep 18 16:17:34.221820 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.221798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86df76cfff-rzhm6" event={"ID":"3b2e70b6-bc6d-441d-8447-91ace8d34925","Type":"ContainerDied","Data":"6a95297c764fc23599bdc189b525f30fb75816f3590bcedbd596128f826f45f0"} Sep 18 16:17:34.221980 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.221827 2564 scope.go:117] "RemoveContainer" containerID="a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2" Sep 18 16:17:34.231742 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.231721 2564 scope.go:117] "RemoveContainer" containerID="a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2" Sep 18 16:17:34.232083 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:17:34.232060 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2\": container with ID starting with a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2 not found: ID does not exist" containerID="a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2" Sep 18 16:17:34.232130 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.232093 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2"} err="failed to get container status \"a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2\": rpc error: code = NotFound desc = could not find container \"a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2\": container with ID starting with a92a41a60502a4c44c2aa1fc3043dd1160b5463f44808c882c050a1c6cdb83f2 not found: ID does not exist" Sep 18 16:17:34.253618 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.253594 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86df76cfff-rzhm6"] Sep 18 16:17:34.259644 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:34.259618 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86df76cfff-rzhm6"] Sep 18 16:17:35.227268 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:35.227236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-d4c7n" event={"ID":"ca6e2edb-271b-43d0-89ee-d0e7a94f1c2f","Type":"ContainerStarted","Data":"25785f307266d4ad04f9898ce2400ee6c0075ccab8f4ab2aca8da71d724f06e0"} Sep 18 16:17:35.638824 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:35.638795 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b2e70b6-bc6d-441d-8447-91ace8d34925" path="/var/lib/kubelet/pods/3b2e70b6-bc6d-441d-8447-91ace8d34925/volumes" Sep 18 16:17:39.245734 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:39.245698 2564 generic.go:356] "Generic (PLEG): container finished" podID="bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af" containerID="ecf02de3549ded48ff93a6d04d9e6fd2bef68f724683871e92b8382b1e8c92e9" exitCode=0 Sep 18 16:17:39.246102 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:39.245774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" event={"ID":"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af","Type":"ContainerDied","Data":"ecf02de3549ded48ff93a6d04d9e6fd2bef68f724683871e92b8382b1e8c92e9"} Sep 18 16:17:39.246102 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:39.246086 2564 scope.go:117] "RemoveContainer" containerID="ecf02de3549ded48ff93a6d04d9e6fd2bef68f724683871e92b8382b1e8c92e9" Sep 18 16:17:40.251544 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:17:40.251510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-wbrs4" event={"ID":"bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af","Type":"ContainerStarted","Data":"ab112408c7601d7a9b7053e1bd0bb11f76f17c8e51d801284127dd763a803f31"} Sep 18 16:18:22.057762 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.057674 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7b5b95d858-mnwjd"] Sep 18 16:18:22.058149 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.057985 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2e70b6-bc6d-441d-8447-91ace8d34925" containerName="registry" Sep 18 16:18:22.058149 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.057998 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2e70b6-bc6d-441d-8447-91ace8d34925" containerName="registry" Sep 18 16:18:22.058149 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.058046 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2e70b6-bc6d-441d-8447-91ace8d34925" containerName="registry" Sep 18 16:18:22.059978 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.059958 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.063739 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.063715 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 18 16:18:22.063858 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.063725 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 18 16:18:22.063858 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.063837 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 18 16:18:22.063979 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.063918 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jqggv\"" Sep 18 16:18:22.064197 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.064161 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 18 16:18:22.068834 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.068811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 18 16:18:22.069827 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.069805 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 18 16:18:22.082237 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.082215 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b5b95d858-mnwjd"] Sep 18 16:18:22.123256 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.123232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9m8k\" (UniqueName: \"kubernetes.io/projected/14f57712-65b1-4721-aff4-bbbe75c87fa9-kube-api-access-g9m8k\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.123355 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.123269 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.123355 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.123286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f57712-65b1-4721-aff4-bbbe75c87fa9-serving-certs-ca-bundle\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.123355 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.123308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-secret-telemeter-client\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.123355 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.123349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-federate-client-tls\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.123516 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.123366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-telemeter-client-tls\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.123516 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.123404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14f57712-65b1-4721-aff4-bbbe75c87fa9-metrics-client-ca\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.123516 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.123430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f57712-65b1-4721-aff4-bbbe75c87fa9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.224105 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.224074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-secret-telemeter-client\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.224214 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.224115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-federate-client-tls\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.224214 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.224136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-telemeter-client-tls\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.224318 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.224245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14f57712-65b1-4721-aff4-bbbe75c87fa9-metrics-client-ca\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.224318 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.224273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f57712-65b1-4721-aff4-bbbe75c87fa9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.224411 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.224322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9m8k\" (UniqueName: \"kubernetes.io/projected/14f57712-65b1-4721-aff4-bbbe75c87fa9-kube-api-access-g9m8k\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.224411 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.224387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.224541 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.224429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f57712-65b1-4721-aff4-bbbe75c87fa9-serving-certs-ca-bundle\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.225040 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.225019 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14f57712-65b1-4721-aff4-bbbe75c87fa9-metrics-client-ca\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.225346 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.225320 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f57712-65b1-4721-aff4-bbbe75c87fa9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.225921 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.225894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f57712-65b1-4721-aff4-bbbe75c87fa9-serving-certs-ca-bundle\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.226552 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.226522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-secret-telemeter-client\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.226853 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.226834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.226936 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.226834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-telemeter-client-tls\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.226984 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.226929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/14f57712-65b1-4721-aff4-bbbe75c87fa9-federate-client-tls\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.232665 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.232648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9m8k\" (UniqueName: \"kubernetes.io/projected/14f57712-65b1-4721-aff4-bbbe75c87fa9-kube-api-access-g9m8k\") pod \"telemeter-client-7b5b95d858-mnwjd\" (UID: \"14f57712-65b1-4721-aff4-bbbe75c87fa9\") " pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.374309 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.374261 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" Sep 18 16:18:22.647588 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:22.647562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b5b95d858-mnwjd"] Sep 18 16:18:22.649002 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:18:22.648973 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f57712_65b1_4721_aff4_bbbe75c87fa9.slice/crio-c1adff05207ca19b8b6e47ecb7cb4bb0bc480e5fe1b20c91421cf05a7a345342 WatchSource:0}: Error finding container c1adff05207ca19b8b6e47ecb7cb4bb0bc480e5fe1b20c91421cf05a7a345342: Status 404 returned error can't find the container with id c1adff05207ca19b8b6e47ecb7cb4bb0bc480e5fe1b20c91421cf05a7a345342 Sep 18 16:18:23.401304 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:23.401270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" event={"ID":"14f57712-65b1-4721-aff4-bbbe75c87fa9","Type":"ContainerStarted","Data":"c1adff05207ca19b8b6e47ecb7cb4bb0bc480e5fe1b20c91421cf05a7a345342"} Sep 18 16:18:25.410314 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:25.410273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" event={"ID":"14f57712-65b1-4721-aff4-bbbe75c87fa9","Type":"ContainerStarted","Data":"8ebda1df9c9e36a044419350974a47e7be6847815648265800327d5782848e3a"} Sep 18 16:18:26.415389 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:26.415353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" event={"ID":"14f57712-65b1-4721-aff4-bbbe75c87fa9","Type":"ContainerStarted","Data":"78465d1fc7fd10925478de5a4776a85cde50510c56eb0661e7ca0ef2517d6de9"} Sep 18 16:18:26.415774 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:26.415452 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" event={"ID":"14f57712-65b1-4721-aff4-bbbe75c87fa9","Type":"ContainerStarted","Data":"20f87d9863ce2b576816e44919487bb1e940b0da7d4bd651be67d9cd084c7099"} Sep 18 16:18:26.439395 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:18:26.439324 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7b5b95d858-mnwjd" podStartSLOduration=1.580596525 podStartE2EDuration="4.439309592s" totalImagesPullingTime="2.858713067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:18:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:18:26.438322819 +0000 UTC m=+172.957590329" watchObservedRunningTime="2026-09-18 16:18:26.439309592 +0000 UTC m=+172.958577090" Sep 18 16:20:35.512979 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:20:35.512558 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:20:35.513872 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:20:35.513850 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:20:35.521943 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:20:35.521919 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:20:35.522747 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:20:35.522724 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:20:35.524991 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:20:35.524973 2564 kubelet.go:1648] "Image garbage collection succeeded" Sep 18 16:21:42.623960 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.623929 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4"] Sep 18 16:21:42.629351 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.627703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" Sep 18 16:21:42.632525 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.632499 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 18 16:21:42.632663 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.632525 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 18 16:21:42.632663 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.632501 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 18 16:21:42.632663 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.632525 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 18 16:21:42.632663 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.632499 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-qrxmr\"" Sep 18 16:21:42.639203 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.639181 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4"] Sep 18 16:21:42.744023 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.743995 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99"] Sep 18 16:21:42.746841 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.746827 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:42.749353 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.749337 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 18 16:21:42.757120 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.757102 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99"] Sep 18 16:21:42.764998 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.764972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhhq\" (UniqueName: \"kubernetes.io/projected/82e01893-5389-45ff-b311-60935618b670-kube-api-access-7lhhq\") pod \"managed-serviceaccount-addon-agent-6c954494bd-2h8t4\" (UID: \"82e01893-5389-45ff-b311-60935618b670\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" Sep 18 16:21:42.765089 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.765042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e01893-5389-45ff-b311-60935618b670-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c954494bd-2h8t4\" (UID: \"82e01893-5389-45ff-b311-60935618b670\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" Sep 18 16:21:42.865896 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.865869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v99sv\" (UniqueName: \"kubernetes.io/projected/93f47aff-2b41-4e1c-88f8-65a2c9fed079-kube-api-access-v99sv\") pod \"klusterlet-addon-workmgr-6579bb9b4d-65l99\" (UID: \"93f47aff-2b41-4e1c-88f8-65a2c9fed079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:42.866041 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.865903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e01893-5389-45ff-b311-60935618b670-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c954494bd-2h8t4\" (UID: \"82e01893-5389-45ff-b311-60935618b670\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" Sep 18 16:21:42.866041 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.865973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/93f47aff-2b41-4e1c-88f8-65a2c9fed079-klusterlet-config\") pod \"klusterlet-addon-workmgr-6579bb9b4d-65l99\" (UID: \"93f47aff-2b41-4e1c-88f8-65a2c9fed079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:42.866041 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.866024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhhq\" (UniqueName: \"kubernetes.io/projected/82e01893-5389-45ff-b311-60935618b670-kube-api-access-7lhhq\") pod \"managed-serviceaccount-addon-agent-6c954494bd-2h8t4\" (UID: \"82e01893-5389-45ff-b311-60935618b670\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" Sep 18 16:21:42.866195 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.866062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f47aff-2b41-4e1c-88f8-65a2c9fed079-tmp\") pod \"klusterlet-addon-workmgr-6579bb9b4d-65l99\" (UID: \"93f47aff-2b41-4e1c-88f8-65a2c9fed079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:42.868415 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.868391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e01893-5389-45ff-b311-60935618b670-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c954494bd-2h8t4\" (UID: \"82e01893-5389-45ff-b311-60935618b670\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" Sep 18 16:21:42.875668 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.875615 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhhq\" (UniqueName: \"kubernetes.io/projected/82e01893-5389-45ff-b311-60935618b670-kube-api-access-7lhhq\") pod \"managed-serviceaccount-addon-agent-6c954494bd-2h8t4\" (UID: \"82e01893-5389-45ff-b311-60935618b670\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" Sep 18 16:21:42.957579 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.957563 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" Sep 18 16:21:42.966719 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.966688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/93f47aff-2b41-4e1c-88f8-65a2c9fed079-klusterlet-config\") pod \"klusterlet-addon-workmgr-6579bb9b4d-65l99\" (UID: \"93f47aff-2b41-4e1c-88f8-65a2c9fed079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:42.966802 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.966728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f47aff-2b41-4e1c-88f8-65a2c9fed079-tmp\") pod \"klusterlet-addon-workmgr-6579bb9b4d-65l99\" (UID: \"93f47aff-2b41-4e1c-88f8-65a2c9fed079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:42.966802 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.966764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v99sv\" (UniqueName: \"kubernetes.io/projected/93f47aff-2b41-4e1c-88f8-65a2c9fed079-kube-api-access-v99sv\") pod \"klusterlet-addon-workmgr-6579bb9b4d-65l99\" (UID: \"93f47aff-2b41-4e1c-88f8-65a2c9fed079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:42.967134 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.967116 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f47aff-2b41-4e1c-88f8-65a2c9fed079-tmp\") pod \"klusterlet-addon-workmgr-6579bb9b4d-65l99\" (UID: \"93f47aff-2b41-4e1c-88f8-65a2c9fed079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:42.968929 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.968909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/93f47aff-2b41-4e1c-88f8-65a2c9fed079-klusterlet-config\") pod \"klusterlet-addon-workmgr-6579bb9b4d-65l99\" (UID: \"93f47aff-2b41-4e1c-88f8-65a2c9fed079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:42.975612 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:42.975586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v99sv\" (UniqueName: \"kubernetes.io/projected/93f47aff-2b41-4e1c-88f8-65a2c9fed079-kube-api-access-v99sv\") pod \"klusterlet-addon-workmgr-6579bb9b4d-65l99\" (UID: \"93f47aff-2b41-4e1c-88f8-65a2c9fed079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:43.054867 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:43.054837 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:43.223588 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:43.223562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4"] Sep 18 16:21:43.225845 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:21:43.225819 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82e01893_5389_45ff_b311_60935618b670.slice/crio-35ee038bb299065a2065aef0ef1a0d38d2d091fffa005bfeb4bf9e88b8d1d63f WatchSource:0}: Error finding container 35ee038bb299065a2065aef0ef1a0d38d2d091fffa005bfeb4bf9e88b8d1d63f: Status 404 returned error can't find the container with id 35ee038bb299065a2065aef0ef1a0d38d2d091fffa005bfeb4bf9e88b8d1d63f Sep 18 16:21:43.227687 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:43.227674 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:21:43.319737 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:43.319714 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99"] Sep 18 16:21:43.323343 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:21:43.323320 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f47aff_2b41_4e1c_88f8_65a2c9fed079.slice/crio-2938e6c07869bfda6668429db859a5065391e987b2f6e97262cdec6e37a1adf1 WatchSource:0}: Error finding container 2938e6c07869bfda6668429db859a5065391e987b2f6e97262cdec6e37a1adf1: Status 404 returned error can't find the container with id 2938e6c07869bfda6668429db859a5065391e987b2f6e97262cdec6e37a1adf1 Sep 18 16:21:44.128548 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:44.128516 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" event={"ID":"82e01893-5389-45ff-b311-60935618b670","Type":"ContainerStarted","Data":"35ee038bb299065a2065aef0ef1a0d38d2d091fffa005bfeb4bf9e88b8d1d63f"} Sep 18 16:21:44.129799 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:44.129771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" event={"ID":"93f47aff-2b41-4e1c-88f8-65a2c9fed079","Type":"ContainerStarted","Data":"2938e6c07869bfda6668429db859a5065391e987b2f6e97262cdec6e37a1adf1"} Sep 18 16:21:48.147257 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:48.147216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" event={"ID":"82e01893-5389-45ff-b311-60935618b670","Type":"ContainerStarted","Data":"2e0e5dea378e6bd7c6d52bced8b62896e74383dd3d35f8527576530d2f235bd9"} Sep 18 16:21:48.148839 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:48.148816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" event={"ID":"93f47aff-2b41-4e1c-88f8-65a2c9fed079","Type":"ContainerStarted","Data":"180e10e86758423c2116b915df5659c9aa98eb806faf19a9cf2f8bfc9d4a0d08"} Sep 18 16:21:48.148942 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:48.148863 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:48.150340 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:48.150321 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" Sep 18 16:21:48.165226 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:48.165188 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c954494bd-2h8t4" podStartSLOduration=1.417056033 podStartE2EDuration="6.165175483s" totalImagesPullingTime="4.74811945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:48.164369807 +0000 UTC m=+374.683637305" watchObservedRunningTime="2026-09-18 16:21:48.165175483 +0000 UTC m=+374.684442982" Sep 18 16:21:48.180029 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:21:48.179984 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6579bb9b4d-65l99" podStartSLOduration=1.5172492370000001 podStartE2EDuration="6.17996943s" totalImagesPullingTime="4.662720193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:48.178945113 +0000 UTC m=+374.698212612" watchObservedRunningTime="2026-09-18 16:21:48.17996943 +0000 UTC m=+374.699236929" Sep 18 16:22:14.649444 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.649410 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-8g6x5"] Sep 18 16:22:14.655466 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.655447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:14.658837 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.658798 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Sep 18 16:22:14.658837 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.658834 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 18 16:22:14.658999 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.658854 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-jrfbn\"" Sep 18 16:22:14.659368 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.659353 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 18 16:22:14.659445 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.659432 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 18 16:22:14.659517 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.659439 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 18 16:22:14.665888 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.665863 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-8g6x5"] Sep 18 16:22:14.700070 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.700043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-certificates\") pod \"keda-operator-579c8958f7-8g6x5\" (UID: \"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05\") " pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:14.700274 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.700253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-cabundle0\") pod \"keda-operator-579c8958f7-8g6x5\" (UID: \"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05\") " pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:14.700318 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.700294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pr9n\" (UniqueName: \"kubernetes.io/projected/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-kube-api-access-8pr9n\") pod \"keda-operator-579c8958f7-8g6x5\" (UID: \"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05\") " pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:14.800764 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.800735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-cabundle0\") pod \"keda-operator-579c8958f7-8g6x5\" (UID: \"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05\") " pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:14.800764 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.800764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pr9n\" (UniqueName: \"kubernetes.io/projected/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-kube-api-access-8pr9n\") pod \"keda-operator-579c8958f7-8g6x5\" (UID: \"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05\") " pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:14.800961 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.800839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-certificates\") pod \"keda-operator-579c8958f7-8g6x5\" (UID: \"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05\") " pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:14.800961 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:22:14.800928 2564 secret.go:281] references non-existent secret key: ca.crt Sep 18 16:22:14.800961 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:22:14.800940 2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Sep 18 16:22:14.800961 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:22:14.800948 2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-8g6x5: references non-existent secret key: ca.crt Sep 18 16:22:14.801092 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:22:14.800996 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-certificates podName:7ab3339b-bc46-4da5-9dc1-749d2d7b0e05 nodeName:}" failed. No retries permitted until 2026-09-18 16:22:15.30098194 +0000 UTC m=+401.820249418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-certificates") pod "keda-operator-579c8958f7-8g6x5" (UID: "7ab3339b-bc46-4da5-9dc1-749d2d7b0e05") : references non-existent secret key: ca.crt Sep 18 16:22:14.801518 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.801472 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-cabundle0\") pod \"keda-operator-579c8958f7-8g6x5\" (UID: \"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05\") " pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:14.816362 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:14.816342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pr9n\" (UniqueName: \"kubernetes.io/projected/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-kube-api-access-8pr9n\") pod \"keda-operator-579c8958f7-8g6x5\" (UID: \"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05\") " pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:15.305302 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:15.305271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-certificates\") pod \"keda-operator-579c8958f7-8g6x5\" (UID: \"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05\") " pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:15.307600 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:15.307578 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ab3339b-bc46-4da5-9dc1-749d2d7b0e05-certificates\") pod \"keda-operator-579c8958f7-8g6x5\" (UID: \"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05\") " pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:15.566269 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:15.566173 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:15.855683 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:22:15.855653 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab3339b_bc46_4da5_9dc1_749d2d7b0e05.slice/crio-128fb03360a199903a47228b2bd287357974c1fa6f8eb843d90d4ff2602e972c WatchSource:0}: Error finding container 128fb03360a199903a47228b2bd287357974c1fa6f8eb843d90d4ff2602e972c: Status 404 returned error can't find the container with id 128fb03360a199903a47228b2bd287357974c1fa6f8eb843d90d4ff2602e972c Sep 18 16:22:15.859246 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:15.859214 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-8g6x5"] Sep 18 16:22:16.249504 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:16.249457 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-8g6x5" event={"ID":"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05","Type":"ContainerStarted","Data":"128fb03360a199903a47228b2bd287357974c1fa6f8eb843d90d4ff2602e972c"} Sep 18 16:22:20.265831 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:20.265789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-8g6x5" event={"ID":"7ab3339b-bc46-4da5-9dc1-749d2d7b0e05","Type":"ContainerStarted","Data":"3b201e13252fc4266aabe1879a537c1374d8ce156f3eadfe076b3e4424dee79f"} Sep 18 16:22:20.266227 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:20.265929 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:22:20.284595 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:20.284547 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-8g6x5" podStartSLOduration=2.883638559 podStartE2EDuration="6.284532805s" totalImagesPullingTime="3.400894246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:20.283934713 +0000 UTC m=+406.803202235" watchObservedRunningTime="2026-09-18 16:22:20.284532805 +0000 UTC m=+406.803800304" Sep 18 16:22:41.272171 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:22:41.272140 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-8g6x5" Sep 18 16:23:32.642726 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.642694 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8"] Sep 18 16:23:32.644790 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.644771 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:23:32.647276 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.647253 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-fzrrf\"" Sep 18 16:23:32.647383 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.647290 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Sep 18 16:23:32.648157 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.648135 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 18 16:23:32.648261 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.648193 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 18 16:23:32.655076 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.655056 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8"] Sep 18 16:23:32.663379 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.663352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-dxfp8\" (UID: \"9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:23:32.663481 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.663416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nbsn\" (UniqueName: \"kubernetes.io/projected/9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5-kube-api-access-7nbsn\") pod \"llmisvc-controller-manager-645f9c8fd6-dxfp8\" (UID: \"9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:23:32.764233 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.764207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-dxfp8\" (UID: \"9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:23:32.764342 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.764269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nbsn\" (UniqueName: \"kubernetes.io/projected/9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5-kube-api-access-7nbsn\") pod \"llmisvc-controller-manager-645f9c8fd6-dxfp8\" (UID: \"9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:23:32.764403 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:23:32.764337 2564 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 18 16:23:32.764403 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:23:32.764394 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5-cert podName:9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5 nodeName:}" failed. No retries permitted until 2026-09-18 16:23:33.26437931 +0000 UTC m=+479.783646787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5-cert") pod "llmisvc-controller-manager-645f9c8fd6-dxfp8" (UID: "9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5") : secret "llmisvc-webhook-server-cert" not found Sep 18 16:23:32.774743 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:32.774722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nbsn\" (UniqueName: \"kubernetes.io/projected/9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5-kube-api-access-7nbsn\") pod \"llmisvc-controller-manager-645f9c8fd6-dxfp8\" (UID: \"9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:23:33.268565 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:33.268534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-dxfp8\" (UID: \"9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:23:33.271077 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:33.271044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-dxfp8\" (UID: \"9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:23:33.554980 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:33.554915 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:23:33.826510 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:33.826421 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8"] Sep 18 16:23:33.829538 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:23:33.829512 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9f2bbe5f_3b2b_4490_8f04_eee23d3db2a5.slice/crio-049dd56ffad3b1a5a2f6487f415e47d3fa7f1dec7612ffc451c21953d3847a8a WatchSource:0}: Error finding container 049dd56ffad3b1a5a2f6487f415e47d3fa7f1dec7612ffc451c21953d3847a8a: Status 404 returned error can't find the container with id 049dd56ffad3b1a5a2f6487f415e47d3fa7f1dec7612ffc451c21953d3847a8a Sep 18 16:23:34.553448 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:34.553408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" event={"ID":"9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5","Type":"ContainerStarted","Data":"049dd56ffad3b1a5a2f6487f415e47d3fa7f1dec7612ffc451c21953d3847a8a"} Sep 18 16:23:36.562146 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:36.562110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" event={"ID":"9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5","Type":"ContainerStarted","Data":"531e59f300ebd88873501041f3fd12c02610f62a94aeedfa90482c14b4501d45"} Sep 18 16:23:36.562502 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:36.562212 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:23:36.581307 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:23:36.581264 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" podStartSLOduration=2.267001303 podStartE2EDuration="4.58125018s" totalImagesPullingTime="2.314248877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:23:36.579262723 +0000 UTC m=+483.098530220" watchObservedRunningTime="2026-09-18 16:23:36.58125018 +0000 UTC m=+483.100517729" Sep 18 16:24:07.569017 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:24:07.568972 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-dxfp8" Sep 18 16:25:35.541163 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:25:35.541122 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:25:35.541664 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:25:35.541607 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:25:35.550340 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:25:35.550316 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:25:35.550566 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:25:35.550550 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:26:37.442650 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.442611 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7"] Sep 18 16:26:37.445877 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.445858 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:37.448423 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.448399 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-191f9-predictor-serving-cert\"" Sep 18 16:26:37.448569 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.448431 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 18 16:26:37.448569 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.448404 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-191f9-kube-rbac-proxy-sar-config\"" Sep 18 16:26:37.449408 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.449391 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 18 16:26:37.449529 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.449419 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z57rx\"" Sep 18 16:26:37.456540 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.456519 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7"] Sep 18 16:26:37.552409 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.552372 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t6jv\" (UniqueName: \"kubernetes.io/projected/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-api-access-2t6jv\") pod \"error-404-isvc-191f9-predictor-86c89bb448-hmcl7\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:37.552409 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.552416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-191f9-predictor-86c89bb448-hmcl7\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:37.552629 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.552443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-proxy-tls\") pod \"error-404-isvc-191f9-predictor-86c89bb448-hmcl7\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:37.653251 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.653223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t6jv\" (UniqueName: \"kubernetes.io/projected/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-api-access-2t6jv\") pod \"error-404-isvc-191f9-predictor-86c89bb448-hmcl7\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:37.653403 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.653260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-191f9-predictor-86c89bb448-hmcl7\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:37.653403 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.653290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-proxy-tls\") pod \"error-404-isvc-191f9-predictor-86c89bb448-hmcl7\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:37.653931 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.653905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-191f9-predictor-86c89bb448-hmcl7\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:37.655659 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.655639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-proxy-tls\") pod \"error-404-isvc-191f9-predictor-86c89bb448-hmcl7\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:37.663301 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.663275 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t6jv\" (UniqueName: \"kubernetes.io/projected/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-api-access-2t6jv\") pod \"error-404-isvc-191f9-predictor-86c89bb448-hmcl7\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:37.755359 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:37.755286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:38.024345 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:38.024314 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7"] Sep 18 16:26:38.026786 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:26:38.026762 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfb1cbe_7e60_4042_9130_a95dd5ee09cb.slice/crio-7ef37a8197b33bd8a6429a3551325ee41fc7043e824e687302ca486a2bc75149 WatchSource:0}: Error finding container 7ef37a8197b33bd8a6429a3551325ee41fc7043e824e687302ca486a2bc75149: Status 404 returned error can't find the container with id 7ef37a8197b33bd8a6429a3551325ee41fc7043e824e687302ca486a2bc75149 Sep 18 16:26:38.253261 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:38.253220 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" event={"ID":"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb","Type":"ContainerStarted","Data":"7ef37a8197b33bd8a6429a3551325ee41fc7043e824e687302ca486a2bc75149"} Sep 18 16:26:49.468807 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:49.468783 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:26:50.313074 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:50.313031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" event={"ID":"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb","Type":"ContainerStarted","Data":"e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45"} Sep 18 16:26:52.323910 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:52.323872 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" event={"ID":"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb","Type":"ContainerStarted","Data":"758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab"} Sep 18 16:26:52.324294 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:52.324144 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:52.324339 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:52.324286 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:52.325699 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:52.325656 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 18 16:26:52.341415 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:52.341370 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podStartSLOduration=1.821355608 podStartE2EDuration="15.341354752s" totalImagesPullingTime="13.519999144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:52.339991763 +0000 UTC m=+678.859259261" watchObservedRunningTime="2026-09-18 16:26:52.341354752 +0000 UTC m=+678.860622251" Sep 18 16:26:53.327693 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:53.327656 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 18 16:26:54.332943 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:54.332896 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 18 16:26:59.337074 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:59.337038 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:26:59.337544 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:26:59.337515 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 18 16:27:09.337891 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:27:09.337842 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 18 16:27:19.337812 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:27:19.337728 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 18 16:27:29.337779 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:27:29.337738 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 18 16:27:39.338662 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:27:39.338626 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:28:18.459360 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.459323 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7"] Sep 18 16:28:18.459917 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.459547 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" containerID="cri-o://e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45" gracePeriod=30 Sep 18 16:28:18.459917 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.459589 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kube-rbac-proxy" containerID="cri-o://758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab" gracePeriod=30 Sep 18 16:28:18.660253 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.660224 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf"] Sep 18 16:28:18.662262 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.662235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:18.664503 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.664468 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8aa04-kube-rbac-proxy-sar-config\"" Sep 18 16:28:18.664632 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.664468 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8aa04-predictor-serving-cert\"" Sep 18 16:28:18.673990 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.673970 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf"] Sep 18 16:28:18.677792 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.677764 2564 generic.go:356] "Generic (PLEG): container finished" podID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerID="758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab" exitCode=2 Sep 18 16:28:18.677890 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.677831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" event={"ID":"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb","Type":"ContainerDied","Data":"758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab"} Sep 18 16:28:18.779464 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.779406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0072adb6-9ad1-44b4-a14c-c15b5c19466c-proxy-tls\") pod \"error-404-isvc-8aa04-predictor-84fd66b648-4gjpf\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:18.779464 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.779446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8aa04-predictor-84fd66b648-4gjpf\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:18.779613 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.779527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqqt\" (UniqueName: \"kubernetes.io/projected/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-api-access-fdqqt\") pod \"error-404-isvc-8aa04-predictor-84fd66b648-4gjpf\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:18.880542 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.880515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqqt\" (UniqueName: \"kubernetes.io/projected/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-api-access-fdqqt\") pod \"error-404-isvc-8aa04-predictor-84fd66b648-4gjpf\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:18.880628 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.880558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0072adb6-9ad1-44b4-a14c-c15b5c19466c-proxy-tls\") pod \"error-404-isvc-8aa04-predictor-84fd66b648-4gjpf\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:18.880691 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.880673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8aa04-predictor-84fd66b648-4gjpf\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:18.881348 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.881331 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8aa04-predictor-84fd66b648-4gjpf\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:18.882835 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.882818 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0072adb6-9ad1-44b4-a14c-c15b5c19466c-proxy-tls\") pod \"error-404-isvc-8aa04-predictor-84fd66b648-4gjpf\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:18.888846 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.888827 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqqt\" (UniqueName: \"kubernetes.io/projected/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-api-access-fdqqt\") pod \"error-404-isvc-8aa04-predictor-84fd66b648-4gjpf\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:18.978589 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:18.978566 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:19.244158 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:19.244128 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf"] Sep 18 16:28:19.246423 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:28:19.246397 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0072adb6_9ad1_44b4_a14c_c15b5c19466c.slice/crio-028c98c6c914a6fc2c6b6c5ab4df11d646225c23d40bcc8f0d3420947e793f22 WatchSource:0}: Error finding container 028c98c6c914a6fc2c6b6c5ab4df11d646225c23d40bcc8f0d3420947e793f22: Status 404 returned error can't find the container with id 028c98c6c914a6fc2c6b6c5ab4df11d646225c23d40bcc8f0d3420947e793f22 Sep 18 16:28:19.333284 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:19.333259 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused" Sep 18 16:28:19.337532 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:19.337514 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 18 16:28:19.682690 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:19.682618 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" event={"ID":"0072adb6-9ad1-44b4-a14c-c15b5c19466c","Type":"ContainerStarted","Data":"7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9"} Sep 18 16:28:19.682690 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:19.682671 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" event={"ID":"0072adb6-9ad1-44b4-a14c-c15b5c19466c","Type":"ContainerStarted","Data":"c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71"} Sep 18 16:28:19.683098 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:19.682696 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" event={"ID":"0072adb6-9ad1-44b4-a14c-c15b5c19466c","Type":"ContainerStarted","Data":"028c98c6c914a6fc2c6b6c5ab4df11d646225c23d40bcc8f0d3420947e793f22"} Sep 18 16:28:19.683098 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:19.682848 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:19.683098 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:19.682889 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:19.684259 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:19.684234 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 18 16:28:19.700224 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:19.700183 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podStartSLOduration=1.7001699829999999 podStartE2EDuration="1.700169983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:19.698960596 +0000 UTC m=+766.218228097" watchObservedRunningTime="2026-09-18 16:28:19.700169983 +0000 UTC m=+766.219437480" Sep 18 16:28:20.686685 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:20.686624 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 18 16:28:21.601645 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.601625 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:28:21.691687 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.691663 2564 generic.go:356] "Generic (PLEG): container finished" podID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerID="e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45" exitCode=0 Sep 18 16:28:21.691989 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.691712 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" Sep 18 16:28:21.691989 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.691725 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" event={"ID":"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb","Type":"ContainerDied","Data":"e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45"} Sep 18 16:28:21.691989 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.691760 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7" event={"ID":"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb","Type":"ContainerDied","Data":"7ef37a8197b33bd8a6429a3551325ee41fc7043e824e687302ca486a2bc75149"} Sep 18 16:28:21.691989 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.691785 2564 scope.go:117] "RemoveContainer" containerID="758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab" Sep 18 16:28:21.698407 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.698386 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t6jv\" (UniqueName: \"kubernetes.io/projected/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-api-access-2t6jv\") pod \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " Sep 18 16:28:21.698534 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.698423 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-proxy-tls\") pod \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " Sep 18 16:28:21.698534 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.698480 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-rbac-proxy-sar-config\") pod \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\" (UID: \"2dfb1cbe-7e60-4042-9130-a95dd5ee09cb\") " Sep 18 16:28:21.698881 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.698848 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" (UID: "2dfb1cbe-7e60-4042-9130-a95dd5ee09cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:28:21.699400 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.699379 2564 scope.go:117] "RemoveContainer" containerID="e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45" Sep 18 16:28:21.700611 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.700590 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-api-access-2t6jv" (OuterVolumeSpecName: "kube-api-access-2t6jv") pod "2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" (UID: "2dfb1cbe-7e60-4042-9130-a95dd5ee09cb"). InnerVolumeSpecName "kube-api-access-2t6jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:28:21.700704 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.700673 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" (UID: "2dfb1cbe-7e60-4042-9130-a95dd5ee09cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:28:21.713259 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.713243 2564 scope.go:117] "RemoveContainer" containerID="758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab" Sep 18 16:28:21.713507 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:28:21.713466 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab\": container with ID starting with 758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab not found: ID does not exist" containerID="758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab" Sep 18 16:28:21.713585 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.713518 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab"} err="failed to get container status \"758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab\": rpc error: code = NotFound desc = could not find container \"758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab\": container with ID starting with 758cb187b5889d3c44bdfe62450c253a5243129de0ba19f5d061c2fc31abaaab not found: ID does not exist" Sep 18 16:28:21.713585 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.713534 2564 scope.go:117] "RemoveContainer" containerID="e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45" Sep 18 16:28:21.713789 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:28:21.713773 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45\": container with ID starting with e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45 not found: ID does not exist" containerID="e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45" Sep 18 16:28:21.713827 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.713795 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45"} err="failed to get container status \"e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45\": rpc error: code = NotFound desc = could not find container \"e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45\": container with ID starting with e5a49c522b9ab1b4cd227e1ea905579cd517254af49125c79536f858f45f9e45 not found: ID does not exist" Sep 18 16:28:21.799442 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.799422 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t6jv\" (UniqueName: \"kubernetes.io/projected/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-api-access-2t6jv\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:28:21.799442 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.799442 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-proxy-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:28:21.799572 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:21.799452 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:28:22.012207 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:22.012179 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7"] Sep 18 16:28:22.014725 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:22.014703 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-191f9-predictor-86c89bb448-hmcl7"] Sep 18 16:28:23.640156 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:23.640120 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" path="/var/lib/kubelet/pods/2dfb1cbe-7e60-4042-9130-a95dd5ee09cb/volumes" Sep 18 16:28:25.691135 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:25.691108 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:28:25.691578 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:25.691552 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 18 16:28:35.691713 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:35.691674 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 18 16:28:45.691547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:45.691507 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 18 16:28:55.692388 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:55.692349 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 18 16:28:59.252472 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.252438 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974"] Sep 18 16:28:59.252868 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.252758 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" Sep 18 16:28:59.252868 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.252768 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" Sep 18 16:28:59.252868 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.252778 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kube-rbac-proxy" Sep 18 16:28:59.252868 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.252783 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kube-rbac-proxy" Sep 18 16:28:59.252868 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.252853 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kube-rbac-proxy" Sep 18 16:28:59.252868 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.252860 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfb1cbe-7e60-4042-9130-a95dd5ee09cb" containerName="kserve-container" Sep 18 16:28:59.255175 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.255155 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.257981 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.257961 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8d780-kube-rbac-proxy-sar-config\"" Sep 18 16:28:59.258322 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.257961 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8d780-predictor-serving-cert\"" Sep 18 16:28:59.266806 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.262341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m828x\" (UniqueName: \"kubernetes.io/projected/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-api-access-m828x\") pod \"error-404-isvc-8d780-predictor-7c7b685fcd-6l974\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.266806 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.262411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8d780-predictor-7c7b685fcd-6l974\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.266806 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.262456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-proxy-tls\") pod \"error-404-isvc-8d780-predictor-7c7b685fcd-6l974\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.271043 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.271017 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974"] Sep 18 16:28:59.363314 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.363289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m828x\" (UniqueName: \"kubernetes.io/projected/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-api-access-m828x\") pod \"error-404-isvc-8d780-predictor-7c7b685fcd-6l974\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.363414 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.363334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8d780-predictor-7c7b685fcd-6l974\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.363414 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.363372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-proxy-tls\") pod \"error-404-isvc-8d780-predictor-7c7b685fcd-6l974\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.363963 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.363944 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8d780-predictor-7c7b685fcd-6l974\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.365691 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.365673 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-proxy-tls\") pod \"error-404-isvc-8d780-predictor-7c7b685fcd-6l974\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.374232 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.374204 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m828x\" (UniqueName: \"kubernetes.io/projected/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-api-access-m828x\") pod \"error-404-isvc-8d780-predictor-7c7b685fcd-6l974\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.566927 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.566873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:28:59.840299 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:28:59.840274 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974"] Sep 18 16:28:59.842673 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:28:59.842628 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f2e7a0_99e3_40c7_9f66_e3069b10a4b9.slice/crio-f3f910361a5da67235af4d1b7e1a9d53ad15f0f2a2143cf80d51295dcb8ac654 WatchSource:0}: Error finding container f3f910361a5da67235af4d1b7e1a9d53ad15f0f2a2143cf80d51295dcb8ac654: Status 404 returned error can't find the container with id f3f910361a5da67235af4d1b7e1a9d53ad15f0f2a2143cf80d51295dcb8ac654 Sep 18 16:29:00.842149 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:00.842113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" event={"ID":"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9","Type":"ContainerStarted","Data":"b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd"} Sep 18 16:29:00.842600 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:00.842170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" event={"ID":"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9","Type":"ContainerStarted","Data":"dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9"} Sep 18 16:29:00.842600 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:00.842196 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:29:00.842600 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:00.842222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" event={"ID":"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9","Type":"ContainerStarted","Data":"f3f910361a5da67235af4d1b7e1a9d53ad15f0f2a2143cf80d51295dcb8ac654"} Sep 18 16:29:00.842600 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:00.842326 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:29:00.843694 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:00.843671 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 18 16:29:00.866051 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:00.866004 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podStartSLOduration=1.865988968 podStartE2EDuration="1.865988968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:29:00.86575343 +0000 UTC m=+807.385020929" watchObservedRunningTime="2026-09-18 16:29:00.865988968 +0000 UTC m=+807.385256471" Sep 18 16:29:01.845837 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:01.845802 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 18 16:29:05.692650 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:05.692612 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:29:06.850803 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:06.850774 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:29:06.851312 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:06.851283 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 18 16:29:16.852047 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:16.852001 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 18 16:29:26.851608 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:26.851566 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 18 16:29:36.851797 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:36.851758 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 18 16:29:46.852618 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:29:46.852582 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:30:35.569933 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:30:35.569859 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:30:35.570427 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:30:35.570067 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:30:35.578027 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:30:35.578005 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:30:35.578131 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:30:35.578056 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:35:35.595245 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:35:35.595209 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:35:35.607383 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:35:35.607356 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:35:35.610150 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:35:35.610129 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:35:35.617406 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:35:35.617385 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:37:43.562699 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.562665 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf"] Sep 18 16:37:43.563217 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.562916 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kube-rbac-proxy" containerID="cri-o://7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9" gracePeriod=30 Sep 18 16:37:43.563217 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.562896 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" containerID="cri-o://c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71" gracePeriod=30 Sep 18 16:37:43.751884 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.751855 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq"] Sep 18 16:37:43.755039 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.755023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:43.757386 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.757350 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6304f-kube-rbac-proxy-sar-config\"" Sep 18 16:37:43.757523 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.757359 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6304f-predictor-serving-cert\"" Sep 18 16:37:43.763916 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.763896 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq"] Sep 18 16:37:43.873806 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.873753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0653d3-39ed-486a-a1cd-1d4ba878feda-proxy-tls\") pod \"error-404-isvc-6304f-predictor-68f6bc4694-98blq\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:43.873806 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.873780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6304f-predictor-68f6bc4694-98blq\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:43.873806 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.873804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s444d\" (UniqueName: \"kubernetes.io/projected/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-api-access-s444d\") pod \"error-404-isvc-6304f-predictor-68f6bc4694-98blq\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:43.903738 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.903712 2564 generic.go:356] "Generic (PLEG): container finished" podID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerID="7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9" exitCode=2 Sep 18 16:37:43.903817 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.903777 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" event={"ID":"0072adb6-9ad1-44b4-a14c-c15b5c19466c","Type":"ContainerDied","Data":"7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9"} Sep 18 16:37:43.974650 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.974628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0653d3-39ed-486a-a1cd-1d4ba878feda-proxy-tls\") pod \"error-404-isvc-6304f-predictor-68f6bc4694-98blq\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:43.974740 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.974653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6304f-predictor-68f6bc4694-98blq\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:43.974740 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.974679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s444d\" (UniqueName: \"kubernetes.io/projected/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-api-access-s444d\") pod \"error-404-isvc-6304f-predictor-68f6bc4694-98blq\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:43.975272 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.975248 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6304f-predictor-68f6bc4694-98blq\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:43.976858 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.976840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0653d3-39ed-486a-a1cd-1d4ba878feda-proxy-tls\") pod \"error-404-isvc-6304f-predictor-68f6bc4694-98blq\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:43.983146 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:43.983123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s444d\" (UniqueName: \"kubernetes.io/projected/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-api-access-s444d\") pod \"error-404-isvc-6304f-predictor-68f6bc4694-98blq\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:44.065581 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:44.065554 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:44.335058 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:44.335035 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq"] Sep 18 16:37:44.337429 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:37:44.337403 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0653d3_39ed_486a_a1cd_1d4ba878feda.slice/crio-027a5050de0ca4882a74a031a144e9968af21c6797344e24902e20b5d569e253 WatchSource:0}: Error finding container 027a5050de0ca4882a74a031a144e9968af21c6797344e24902e20b5d569e253: Status 404 returned error can't find the container with id 027a5050de0ca4882a74a031a144e9968af21c6797344e24902e20b5d569e253 Sep 18 16:37:44.910126 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:44.910094 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" event={"ID":"dc0653d3-39ed-486a-a1cd-1d4ba878feda","Type":"ContainerStarted","Data":"47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258"} Sep 18 16:37:44.910473 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:44.910146 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" event={"ID":"dc0653d3-39ed-486a-a1cd-1d4ba878feda","Type":"ContainerStarted","Data":"9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9"} Sep 18 16:37:44.910473 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:44.910168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" event={"ID":"dc0653d3-39ed-486a-a1cd-1d4ba878feda","Type":"ContainerStarted","Data":"027a5050de0ca4882a74a031a144e9968af21c6797344e24902e20b5d569e253"} Sep 18 16:37:44.910473 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:44.910191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:44.910473 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:44.910210 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:44.911527 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:44.911504 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 18 16:37:44.929853 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:44.929812 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" podStartSLOduration=1.9298004629999999 podStartE2EDuration="1.929800463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:37:44.927839764 +0000 UTC m=+1331.447107265" watchObservedRunningTime="2026-09-18 16:37:44.929800463 +0000 UTC m=+1331.449067962" Sep 18 16:37:45.687362 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:45.687324 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused" Sep 18 16:37:45.691716 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:45.691689 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 18 16:37:45.919191 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:45.919147 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 18 16:37:46.501020 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.500998 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:37:46.595981 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.595948 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0072adb6-9ad1-44b4-a14c-c15b5c19466c-proxy-tls\") pod \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " Sep 18 16:37:46.596119 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.596022 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdqqt\" (UniqueName: \"kubernetes.io/projected/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-api-access-fdqqt\") pod \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " Sep 18 16:37:46.596189 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.596149 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-rbac-proxy-sar-config\") pod \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\" (UID: \"0072adb6-9ad1-44b4-a14c-c15b5c19466c\") " Sep 18 16:37:46.596543 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.596514 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0072adb6-9ad1-44b4-a14c-c15b5c19466c" (UID: "0072adb6-9ad1-44b4-a14c-c15b5c19466c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:37:46.597883 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.597862 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0072adb6-9ad1-44b4-a14c-c15b5c19466c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0072adb6-9ad1-44b4-a14c-c15b5c19466c" (UID: "0072adb6-9ad1-44b4-a14c-c15b5c19466c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:37:46.597994 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.597976 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-api-access-fdqqt" (OuterVolumeSpecName: "kube-api-access-fdqqt") pod "0072adb6-9ad1-44b4-a14c-c15b5c19466c" (UID: "0072adb6-9ad1-44b4-a14c-c15b5c19466c"). InnerVolumeSpecName "kube-api-access-fdqqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:37:46.697264 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.697242 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0072adb6-9ad1-44b4-a14c-c15b5c19466c-proxy-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:37:46.697264 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.697263 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdqqt\" (UniqueName: \"kubernetes.io/projected/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-api-access-fdqqt\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:37:46.697386 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.697272 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0072adb6-9ad1-44b4-a14c-c15b5c19466c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:37:46.924905 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.924807 2564 generic.go:356] "Generic (PLEG): container finished" podID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerID="c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71" exitCode=0 Sep 18 16:37:46.924905 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.924858 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" Sep 18 16:37:46.924905 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.924866 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" event={"ID":"0072adb6-9ad1-44b4-a14c-c15b5c19466c","Type":"ContainerDied","Data":"c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71"} Sep 18 16:37:46.925395 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.924923 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf" event={"ID":"0072adb6-9ad1-44b4-a14c-c15b5c19466c","Type":"ContainerDied","Data":"028c98c6c914a6fc2c6b6c5ab4df11d646225c23d40bcc8f0d3420947e793f22"} Sep 18 16:37:46.925395 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.924962 2564 scope.go:117] "RemoveContainer" containerID="7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9" Sep 18 16:37:46.934435 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.934417 2564 scope.go:117] "RemoveContainer" containerID="c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71" Sep 18 16:37:46.941544 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.941521 2564 scope.go:117] "RemoveContainer" containerID="7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9" Sep 18 16:37:46.941811 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:37:46.941793 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9\": container with ID starting with 7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9 not found: ID does not exist" containerID="7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9" Sep 18 16:37:46.941869 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.941817 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9"} err="failed to get container status \"7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9\": rpc error: code = NotFound desc = could not find container \"7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9\": container with ID starting with 7012505812fede2a40572e900ae8b7cbb5269ee5ffca006ae4d39717ad7ecdc9 not found: ID does not exist" Sep 18 16:37:46.941869 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.941832 2564 scope.go:117] "RemoveContainer" containerID="c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71" Sep 18 16:37:46.942087 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:37:46.942068 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71\": container with ID starting with c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71 not found: ID does not exist" containerID="c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71" Sep 18 16:37:46.942121 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.942092 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71"} err="failed to get container status \"c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71\": rpc error: code = NotFound desc = could not find container \"c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71\": container with ID starting with c6228e342c709625aa3b58aa1ad2341eff9e64628c703df813df4c3ce7e80b71 not found: ID does not exist" Sep 18 16:37:46.948934 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.948912 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf"] Sep 18 16:37:46.953301 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:46.953281 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8aa04-predictor-84fd66b648-4gjpf"] Sep 18 16:37:47.639394 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:47.639362 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" path="/var/lib/kubelet/pods/0072adb6-9ad1-44b4-a14c-c15b5c19466c/volumes" Sep 18 16:37:50.924105 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:50.924073 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:37:50.924626 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:37:50.924548 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 18 16:38:00.925215 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:00.925177 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 18 16:38:10.924644 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:10.924603 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 18 16:38:20.925354 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:20.925316 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 18 16:38:24.691012 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.690981 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974"] Sep 18 16:38:24.691417 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.691210 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" containerID="cri-o://dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9" gracePeriod=30 Sep 18 16:38:24.691417 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.691254 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kube-rbac-proxy" containerID="cri-o://b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd" gracePeriod=30 Sep 18 16:38:24.938674 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.938626 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl"] Sep 18 16:38:24.938977 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.938964 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" Sep 18 16:38:24.939019 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.938980 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" Sep 18 16:38:24.939019 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.938994 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kube-rbac-proxy" Sep 18 16:38:24.939019 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.939000 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kube-rbac-proxy" Sep 18 16:38:24.939109 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.939074 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kserve-container" Sep 18 16:38:24.939109 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.939082 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0072adb6-9ad1-44b4-a14c-c15b5c19466c" containerName="kube-rbac-proxy" Sep 18 16:38:24.942018 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.941980 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:24.944387 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.944364 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7c9ed-kube-rbac-proxy-sar-config\"" Sep 18 16:38:24.944534 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.944364 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7c9ed-predictor-serving-cert\"" Sep 18 16:38:24.952950 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:24.952926 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl"] Sep 18 16:38:25.069327 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.069291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c533cc1e-39ac-437b-b37b-4720c41e4f3e-proxy-tls\") pod \"error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:25.069462 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.069341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:25.069462 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.069419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwsp9\" (UniqueName: \"kubernetes.io/projected/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-api-access-kwsp9\") pod \"error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:25.074051 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.074025 2564 generic.go:356] "Generic (PLEG): container finished" podID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerID="b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd" exitCode=2 Sep 18 16:38:25.074138 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.074064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" event={"ID":"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9","Type":"ContainerDied","Data":"b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd"} Sep 18 16:38:25.170198 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.170173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c533cc1e-39ac-437b-b37b-4720c41e4f3e-proxy-tls\") pod \"error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:25.170305 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.170218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:25.170305 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.170253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwsp9\" (UniqueName: \"kubernetes.io/projected/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-api-access-kwsp9\") pod \"error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:25.170405 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:38:25.170308 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-serving-cert: secret "error-404-isvc-7c9ed-predictor-serving-cert" not found Sep 18 16:38:25.170405 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:38:25.170367 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c533cc1e-39ac-437b-b37b-4720c41e4f3e-proxy-tls podName:c533cc1e-39ac-437b-b37b-4720c41e4f3e nodeName:}" failed. No retries permitted until 2026-09-18 16:38:25.670351665 +0000 UTC m=+1372.189619157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c533cc1e-39ac-437b-b37b-4720c41e4f3e-proxy-tls") pod "error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" (UID: "c533cc1e-39ac-437b-b37b-4720c41e4f3e") : secret "error-404-isvc-7c9ed-predictor-serving-cert" not found Sep 18 16:38:25.170939 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.170921 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:25.181068 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.181052 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwsp9\" (UniqueName: \"kubernetes.io/projected/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-api-access-kwsp9\") pod \"error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:25.672714 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.672685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c533cc1e-39ac-437b-b37b-4720c41e4f3e-proxy-tls\") pod \"error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:25.675032 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.675000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c533cc1e-39ac-437b-b37b-4720c41e4f3e-proxy-tls\") pod \"error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:25.855392 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:25.855359 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:26.130119 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:26.129961 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl"] Sep 18 16:38:26.132257 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:38:26.132229 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc533cc1e_39ac_437b_b37b_4720c41e4f3e.slice/crio-393320e3de7fbf3faebe84d489681a11dace17ef50dc353c9bcbcb83ce01e064 WatchSource:0}: Error finding container 393320e3de7fbf3faebe84d489681a11dace17ef50dc353c9bcbcb83ce01e064: Status 404 returned error can't find the container with id 393320e3de7fbf3faebe84d489681a11dace17ef50dc353c9bcbcb83ce01e064 Sep 18 16:38:26.846076 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:26.846027 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Sep 18 16:38:26.851422 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:26.851377 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 18 16:38:27.082384 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.082348 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" event={"ID":"c533cc1e-39ac-437b-b37b-4720c41e4f3e","Type":"ContainerStarted","Data":"134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c"} Sep 18 16:38:27.082778 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.082404 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" event={"ID":"c533cc1e-39ac-437b-b37b-4720c41e4f3e","Type":"ContainerStarted","Data":"f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe"} Sep 18 16:38:27.082778 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.082429 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:27.082778 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.082446 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" event={"ID":"c533cc1e-39ac-437b-b37b-4720c41e4f3e","Type":"ContainerStarted","Data":"393320e3de7fbf3faebe84d489681a11dace17ef50dc353c9bcbcb83ce01e064"} Sep 18 16:38:27.101401 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.101316 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" podStartSLOduration=3.101302203 podStartE2EDuration="3.101302203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:38:27.099963634 +0000 UTC m=+1373.619231131" watchObservedRunningTime="2026-09-18 16:38:27.101302203 +0000 UTC m=+1373.620569740" Sep 18 16:38:27.634967 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.634940 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:38:27.788313 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.788240 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-rbac-proxy-sar-config\") pod \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " Sep 18 16:38:27.788313 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.788299 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-proxy-tls\") pod \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " Sep 18 16:38:27.788527 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.788348 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m828x\" (UniqueName: \"kubernetes.io/projected/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-api-access-m828x\") pod \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\" (UID: \"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9\") " Sep 18 16:38:27.788676 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.788640 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" (UID: "55f2e7a0-99e3-40c7-9f66-e3069b10a4b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:38:27.790292 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.790265 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-api-access-m828x" (OuterVolumeSpecName: "kube-api-access-m828x") pod "55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" (UID: "55f2e7a0-99e3-40c7-9f66-e3069b10a4b9"). InnerVolumeSpecName "kube-api-access-m828x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:38:27.790384 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.790267 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" (UID: "55f2e7a0-99e3-40c7-9f66-e3069b10a4b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:38:27.889748 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.889723 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-proxy-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:38:27.889748 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.889749 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m828x\" (UniqueName: \"kubernetes.io/projected/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-api-access-m828x\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:38:27.889863 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:27.889757 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:38:28.088276 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.088214 2564 generic.go:356] "Generic (PLEG): container finished" podID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerID="dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9" exitCode=0 Sep 18 16:38:28.088276 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.088269 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" Sep 18 16:38:28.088679 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.088298 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" event={"ID":"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9","Type":"ContainerDied","Data":"dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9"} Sep 18 16:38:28.088679 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.088356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974" event={"ID":"55f2e7a0-99e3-40c7-9f66-e3069b10a4b9","Type":"ContainerDied","Data":"f3f910361a5da67235af4d1b7e1a9d53ad15f0f2a2143cf80d51295dcb8ac654"} Sep 18 16:38:28.088679 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.088384 2564 scope.go:117] "RemoveContainer" containerID="b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd" Sep 18 16:38:28.088847 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.088814 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:28.090381 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.090348 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 18 16:38:28.096952 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.096930 2564 scope.go:117] "RemoveContainer" containerID="dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9" Sep 18 16:38:28.104442 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.104422 2564 scope.go:117] "RemoveContainer" containerID="b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd" Sep 18 16:38:28.104715 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:38:28.104697 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd\": container with ID starting with b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd not found: ID does not exist" containerID="b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd" Sep 18 16:38:28.104763 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.104723 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd"} err="failed to get container status \"b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd\": rpc error: code = NotFound desc = could not find container \"b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd\": container with ID starting with b8fa3c79a489ffd3b3488b187b12898056b288f65f57622def950cf759e511dd not found: ID does not exist" Sep 18 16:38:28.104763 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.104740 2564 scope.go:117] "RemoveContainer" containerID="dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9" Sep 18 16:38:28.104991 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:38:28.104976 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9\": container with ID starting with dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9 not found: ID does not exist" containerID="dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9" Sep 18 16:38:28.105043 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.104992 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9"} err="failed to get container status \"dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9\": rpc error: code = NotFound desc = could not find container \"dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9\": container with ID starting with dcaa987ad0e07d037e29e111b3d040c0dd71317b72444fa89262890f6860b8d9 not found: ID does not exist" Sep 18 16:38:28.114227 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.114206 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974"] Sep 18 16:38:28.118162 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:28.118144 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d780-predictor-7c7b685fcd-6l974"] Sep 18 16:38:29.093437 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:29.093391 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 18 16:38:29.640026 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:29.639992 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" path="/var/lib/kubelet/pods/55f2e7a0-99e3-40c7-9f66-e3069b10a4b9/volumes" Sep 18 16:38:30.925291 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:30.925264 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:38:34.097770 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:34.097744 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:38:34.098181 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:34.098151 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 18 16:38:44.099076 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:44.099037 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 18 16:38:54.098952 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:38:54.098909 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 18 16:39:04.098522 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.098471 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 18 16:39:04.392776 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.392702 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq"] Sep 18 16:39:04.393029 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.392976 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kserve-container" containerID="cri-o://9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9" gracePeriod=30 Sep 18 16:39:04.393112 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.393002 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kube-rbac-proxy" containerID="cri-o://47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258" gracePeriod=30 Sep 18 16:39:04.547413 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.547385 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk"] Sep 18 16:39:04.547714 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.547702 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" Sep 18 16:39:04.547761 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.547715 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" Sep 18 16:39:04.547761 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.547729 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kube-rbac-proxy" Sep 18 16:39:04.547761 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.547734 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kube-rbac-proxy" Sep 18 16:39:04.547856 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.547783 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kube-rbac-proxy" Sep 18 16:39:04.547856 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.547792 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f2e7a0-99e3-40c7-9f66-e3069b10a4b9" containerName="kserve-container" Sep 18 16:39:04.550677 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.550659 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:04.553091 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.553060 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-de495-kube-rbac-proxy-sar-config\"" Sep 18 16:39:04.553091 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.553060 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-de495-predictor-serving-cert\"" Sep 18 16:39:04.561379 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.561358 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk"] Sep 18 16:39:04.648266 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.648215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb03191d-d415-46ec-9ae0-bd9af3340e6c-proxy-tls\") pod \"error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:04.648266 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.648248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8kg2\" (UniqueName: \"kubernetes.io/projected/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-api-access-p8kg2\") pod \"error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:04.648417 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.648277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:04.749414 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.749390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb03191d-d415-46ec-9ae0-bd9af3340e6c-proxy-tls\") pod \"error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:04.749527 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.749415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8kg2\" (UniqueName: \"kubernetes.io/projected/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-api-access-p8kg2\") pod \"error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:04.749527 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.749445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:04.749648 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:39:04.749559 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-de495-predictor-serving-cert: secret "error-404-isvc-de495-predictor-serving-cert" not found Sep 18 16:39:04.749648 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:39:04.749632 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb03191d-d415-46ec-9ae0-bd9af3340e6c-proxy-tls podName:fb03191d-d415-46ec-9ae0-bd9af3340e6c nodeName:}" failed. No retries permitted until 2026-09-18 16:39:05.249602639 +0000 UTC m=+1411.768870128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb03191d-d415-46ec-9ae0-bd9af3340e6c-proxy-tls") pod "error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" (UID: "fb03191d-d415-46ec-9ae0-bd9af3340e6c") : secret "error-404-isvc-de495-predictor-serving-cert" not found Sep 18 16:39:04.750017 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.749997 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:04.759667 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:04.759635 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8kg2\" (UniqueName: \"kubernetes.io/projected/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-api-access-p8kg2\") pod \"error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:05.236248 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:05.236212 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerID="47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258" exitCode=2 Sep 18 16:39:05.236630 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:05.236294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" event={"ID":"dc0653d3-39ed-486a-a1cd-1d4ba878feda","Type":"ContainerDied","Data":"47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258"} Sep 18 16:39:05.254310 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:05.254279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb03191d-d415-46ec-9ae0-bd9af3340e6c-proxy-tls\") pod \"error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:05.256559 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:05.256539 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb03191d-d415-46ec-9ae0-bd9af3340e6c-proxy-tls\") pod \"error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:05.460095 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:05.460067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:05.731377 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:05.731353 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk"] Sep 18 16:39:05.733904 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:39:05.733873 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb03191d_d415_46ec_9ae0_bd9af3340e6c.slice/crio-6650c0d28554c01ccd0109e600d2624227e9138889b3f6d3b4057d61e613bfd1 WatchSource:0}: Error finding container 6650c0d28554c01ccd0109e600d2624227e9138889b3f6d3b4057d61e613bfd1: Status 404 returned error can't find the container with id 6650c0d28554c01ccd0109e600d2624227e9138889b3f6d3b4057d61e613bfd1 Sep 18 16:39:05.919304 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:05.919276 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 18 16:39:06.241412 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:06.241376 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" event={"ID":"fb03191d-d415-46ec-9ae0-bd9af3340e6c","Type":"ContainerStarted","Data":"0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440"} Sep 18 16:39:06.241773 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:06.241441 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:06.241773 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:06.241462 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" event={"ID":"fb03191d-d415-46ec-9ae0-bd9af3340e6c","Type":"ContainerStarted","Data":"3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae"} Sep 18 16:39:06.241773 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:06.241510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:06.241773 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:06.241530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" event={"ID":"fb03191d-d415-46ec-9ae0-bd9af3340e6c","Type":"ContainerStarted","Data":"6650c0d28554c01ccd0109e600d2624227e9138889b3f6d3b4057d61e613bfd1"} Sep 18 16:39:06.242982 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:06.242957 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 18 16:39:06.261433 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:06.261390 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podStartSLOduration=2.261375682 podStartE2EDuration="2.261375682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:39:06.259805771 +0000 UTC m=+1412.779073269" watchObservedRunningTime="2026-09-18 16:39:06.261375682 +0000 UTC m=+1412.780643179" Sep 18 16:39:07.245899 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.245859 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 18 16:39:07.532431 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.532409 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:39:07.573826 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.573807 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s444d\" (UniqueName: \"kubernetes.io/projected/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-api-access-s444d\") pod \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " Sep 18 16:39:07.573928 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.573839 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-rbac-proxy-sar-config\") pod \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " Sep 18 16:39:07.573928 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.573872 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0653d3-39ed-486a-a1cd-1d4ba878feda-proxy-tls\") pod \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\" (UID: \"dc0653d3-39ed-486a-a1cd-1d4ba878feda\") " Sep 18 16:39:07.574115 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.574094 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc0653d3-39ed-486a-a1cd-1d4ba878feda" (UID: "dc0653d3-39ed-486a-a1cd-1d4ba878feda"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:39:07.575835 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.575790 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc0653d3-39ed-486a-a1cd-1d4ba878feda-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc0653d3-39ed-486a-a1cd-1d4ba878feda" (UID: "dc0653d3-39ed-486a-a1cd-1d4ba878feda"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:39:07.575914 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.575859 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-api-access-s444d" (OuterVolumeSpecName: "kube-api-access-s444d") pod "dc0653d3-39ed-486a-a1cd-1d4ba878feda" (UID: "dc0653d3-39ed-486a-a1cd-1d4ba878feda"). InnerVolumeSpecName "kube-api-access-s444d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:39:07.674498 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.674464 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s444d\" (UniqueName: \"kubernetes.io/projected/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-api-access-s444d\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:39:07.674569 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.674502 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0653d3-39ed-486a-a1cd-1d4ba878feda-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:39:07.674569 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:07.674512 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0653d3-39ed-486a-a1cd-1d4ba878feda-proxy-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:39:08.251951 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.251913 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerID="9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9" exitCode=0 Sep 18 16:39:08.252383 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.251967 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" Sep 18 16:39:08.252383 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.252006 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" event={"ID":"dc0653d3-39ed-486a-a1cd-1d4ba878feda","Type":"ContainerDied","Data":"9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9"} Sep 18 16:39:08.252383 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.252068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq" event={"ID":"dc0653d3-39ed-486a-a1cd-1d4ba878feda","Type":"ContainerDied","Data":"027a5050de0ca4882a74a031a144e9968af21c6797344e24902e20b5d569e253"} Sep 18 16:39:08.252383 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.252095 2564 scope.go:117] "RemoveContainer" containerID="47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258" Sep 18 16:39:08.260811 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.260794 2564 scope.go:117] "RemoveContainer" containerID="9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9" Sep 18 16:39:08.268200 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.268184 2564 scope.go:117] "RemoveContainer" containerID="47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258" Sep 18 16:39:08.268462 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:39:08.268443 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258\": container with ID starting with 47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258 not found: ID does not exist" containerID="47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258" Sep 18 16:39:08.268556 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.268473 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258"} err="failed to get container status \"47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258\": rpc error: code = NotFound desc = could not find container \"47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258\": container with ID starting with 47813fd22fd21ace71dee38e9443c017b5ae1e655a73d2b3e16fc4a772ba7258 not found: ID does not exist" Sep 18 16:39:08.268556 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.268509 2564 scope.go:117] "RemoveContainer" containerID="9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9" Sep 18 16:39:08.268770 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:39:08.268752 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9\": container with ID starting with 9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9 not found: ID does not exist" containerID="9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9" Sep 18 16:39:08.268810 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.268775 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9"} err="failed to get container status \"9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9\": rpc error: code = NotFound desc = could not find container \"9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9\": container with ID starting with 9b1386821b8980101a62885d2e503bba697ba6dea5fd20acafcf6fdd46f960b9 not found: ID does not exist" Sep 18 16:39:08.272441 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.272420 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq"] Sep 18 16:39:08.274269 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:08.274249 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6304f-predictor-68f6bc4694-98blq"] Sep 18 16:39:09.644977 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:09.644939 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" path="/var/lib/kubelet/pods/dc0653d3-39ed-486a-a1cd-1d4ba878feda/volumes" Sep 18 16:39:12.251614 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:12.251580 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:12.252301 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:12.252267 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 18 16:39:14.099533 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:14.099504 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:39:22.252185 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:22.252084 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 18 16:39:32.252553 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:32.252517 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 18 16:39:42.252977 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:42.252935 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 18 16:39:45.814344 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:45.814313 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl"] Sep 18 16:39:45.814746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:45.814586 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kserve-container" containerID="cri-o://f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe" gracePeriod=30 Sep 18 16:39:45.814746 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:45.814624 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kube-rbac-proxy" containerID="cri-o://134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c" gracePeriod=30 Sep 18 16:39:46.021244 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.021219 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb"] Sep 18 16:39:46.021536 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.021522 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kube-rbac-proxy" Sep 18 16:39:46.021581 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.021537 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kube-rbac-proxy" Sep 18 16:39:46.021581 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.021550 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kserve-container" Sep 18 16:39:46.021581 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.021556 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kserve-container" Sep 18 16:39:46.021670 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.021607 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kserve-container" Sep 18 16:39:46.021670 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.021617 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc0653d3-39ed-486a-a1cd-1d4ba878feda" containerName="kube-rbac-proxy" Sep 18 16:39:46.024431 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.024416 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.026916 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.026898 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-630d7-predictor-serving-cert\"" Sep 18 16:39:46.027146 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.027130 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-630d7-kube-rbac-proxy-sar-config\"" Sep 18 16:39:46.037615 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.037596 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb"] Sep 18 16:39:46.137649 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.137593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1518096-00eb-41fb-96e4-d54adf968795-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-630d7-predictor-7bc45d9448-6gblb\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.137649 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.137642 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1518096-00eb-41fb-96e4-d54adf968795-proxy-tls\") pod \"error-404-isvc-630d7-predictor-7bc45d9448-6gblb\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.137817 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.137737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgvb5\" (UniqueName: \"kubernetes.io/projected/d1518096-00eb-41fb-96e4-d54adf968795-kube-api-access-fgvb5\") pod \"error-404-isvc-630d7-predictor-7bc45d9448-6gblb\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.238776 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.238749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1518096-00eb-41fb-96e4-d54adf968795-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-630d7-predictor-7bc45d9448-6gblb\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.238883 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.238788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1518096-00eb-41fb-96e4-d54adf968795-proxy-tls\") pod \"error-404-isvc-630d7-predictor-7bc45d9448-6gblb\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.238883 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.238824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgvb5\" (UniqueName: \"kubernetes.io/projected/d1518096-00eb-41fb-96e4-d54adf968795-kube-api-access-fgvb5\") pod \"error-404-isvc-630d7-predictor-7bc45d9448-6gblb\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.238999 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:39:46.238916 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-630d7-predictor-serving-cert: secret "error-404-isvc-630d7-predictor-serving-cert" not found Sep 18 16:39:46.238999 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:39:46.238981 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1518096-00eb-41fb-96e4-d54adf968795-proxy-tls podName:d1518096-00eb-41fb-96e4-d54adf968795 nodeName:}" failed. No retries permitted until 2026-09-18 16:39:46.738963641 +0000 UTC m=+1453.258231132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1518096-00eb-41fb-96e4-d54adf968795-proxy-tls") pod "error-404-isvc-630d7-predictor-7bc45d9448-6gblb" (UID: "d1518096-00eb-41fb-96e4-d54adf968795") : secret "error-404-isvc-630d7-predictor-serving-cert" not found Sep 18 16:39:46.239432 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.239408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1518096-00eb-41fb-96e4-d54adf968795-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-630d7-predictor-7bc45d9448-6gblb\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.247495 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.247458 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgvb5\" (UniqueName: \"kubernetes.io/projected/d1518096-00eb-41fb-96e4-d54adf968795-kube-api-access-fgvb5\") pod \"error-404-isvc-630d7-predictor-7bc45d9448-6gblb\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.418103 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.418038 2564 generic.go:356] "Generic (PLEG): container finished" podID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerID="134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c" exitCode=2 Sep 18 16:39:46.418196 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.418111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" event={"ID":"c533cc1e-39ac-437b-b37b-4720c41e4f3e","Type":"ContainerDied","Data":"134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c"} Sep 18 16:39:46.743886 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.743856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1518096-00eb-41fb-96e4-d54adf968795-proxy-tls\") pod \"error-404-isvc-630d7-predictor-7bc45d9448-6gblb\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.746164 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.746149 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1518096-00eb-41fb-96e4-d54adf968795-proxy-tls\") pod \"error-404-isvc-630d7-predictor-7bc45d9448-6gblb\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:46.933547 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:46.933515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:47.205426 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:47.205365 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb"] Sep 18 16:39:47.208061 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:39:47.208031 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1518096_00eb_41fb_96e4_d54adf968795.slice/crio-1d13e489e5c3e1dc04db6aa0aca70d1c7f3d37c3efa622ddaf372cbf7ec5e9a8 WatchSource:0}: Error finding container 1d13e489e5c3e1dc04db6aa0aca70d1c7f3d37c3efa622ddaf372cbf7ec5e9a8: Status 404 returned error can't find the container with id 1d13e489e5c3e1dc04db6aa0aca70d1c7f3d37c3efa622ddaf372cbf7ec5e9a8 Sep 18 16:39:47.428846 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:47.428812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" event={"ID":"d1518096-00eb-41fb-96e4-d54adf968795","Type":"ContainerStarted","Data":"163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37"} Sep 18 16:39:47.428987 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:47.428894 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" event={"ID":"d1518096-00eb-41fb-96e4-d54adf968795","Type":"ContainerStarted","Data":"dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06"} Sep 18 16:39:47.428987 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:47.428922 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:47.428987 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:47.428938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" event={"ID":"d1518096-00eb-41fb-96e4-d54adf968795","Type":"ContainerStarted","Data":"1d13e489e5c3e1dc04db6aa0aca70d1c7f3d37c3efa622ddaf372cbf7ec5e9a8"} Sep 18 16:39:47.444471 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:47.444428 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podStartSLOduration=2.44441515 podStartE2EDuration="2.44441515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:39:47.444013844 +0000 UTC m=+1453.963281341" watchObservedRunningTime="2026-09-18 16:39:47.44441515 +0000 UTC m=+1453.963682647" Sep 18 16:39:48.432758 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:48.432722 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:48.433979 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:48.433950 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 18 16:39:48.860898 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:48.860875 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:39:48.957821 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:48.957789 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c533cc1e-39ac-437b-b37b-4720c41e4f3e-proxy-tls\") pod \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " Sep 18 16:39:48.957821 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:48.957820 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwsp9\" (UniqueName: \"kubernetes.io/projected/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-api-access-kwsp9\") pod \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " Sep 18 16:39:48.958016 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:48.957892 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-rbac-proxy-sar-config\") pod \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\" (UID: \"c533cc1e-39ac-437b-b37b-4720c41e4f3e\") " Sep 18 16:39:48.958384 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:48.958338 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c533cc1e-39ac-437b-b37b-4720c41e4f3e" (UID: "c533cc1e-39ac-437b-b37b-4720c41e4f3e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:39:48.960132 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:48.960097 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-api-access-kwsp9" (OuterVolumeSpecName: "kube-api-access-kwsp9") pod "c533cc1e-39ac-437b-b37b-4720c41e4f3e" (UID: "c533cc1e-39ac-437b-b37b-4720c41e4f3e"). InnerVolumeSpecName "kube-api-access-kwsp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:39:48.960320 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:48.960295 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c533cc1e-39ac-437b-b37b-4720c41e4f3e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c533cc1e-39ac-437b-b37b-4720c41e4f3e" (UID: "c533cc1e-39ac-437b-b37b-4720c41e4f3e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:39:49.058841 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.058795 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:39:49.058841 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.058816 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c533cc1e-39ac-437b-b37b-4720c41e4f3e-proxy-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:39:49.058841 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.058827 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwsp9\" (UniqueName: \"kubernetes.io/projected/c533cc1e-39ac-437b-b37b-4720c41e4f3e-kube-api-access-kwsp9\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:39:49.437721 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.437696 2564 generic.go:356] "Generic (PLEG): container finished" podID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerID="f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe" exitCode=0 Sep 18 16:39:49.438056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.437748 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" Sep 18 16:39:49.438056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.437781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" event={"ID":"c533cc1e-39ac-437b-b37b-4720c41e4f3e","Type":"ContainerDied","Data":"f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe"} Sep 18 16:39:49.438056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.437847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl" event={"ID":"c533cc1e-39ac-437b-b37b-4720c41e4f3e","Type":"ContainerDied","Data":"393320e3de7fbf3faebe84d489681a11dace17ef50dc353c9bcbcb83ce01e064"} Sep 18 16:39:49.438056 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.437875 2564 scope.go:117] "RemoveContainer" containerID="134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c" Sep 18 16:39:49.438429 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.438403 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 18 16:39:49.446668 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.446633 2564 scope.go:117] "RemoveContainer" containerID="f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe" Sep 18 16:39:49.455072 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.455049 2564 scope.go:117] "RemoveContainer" containerID="134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c" Sep 18 16:39:49.455354 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:39:49.455335 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c\": container with ID starting with 134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c not found: ID does not exist" containerID="134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c" Sep 18 16:39:49.455423 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.455361 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c"} err="failed to get container status \"134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c\": rpc error: code = NotFound desc = could not find container \"134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c\": container with ID starting with 134529811f3316889135b7ebe0afccde13ecdc709473ca94d4a6238bba27e10c not found: ID does not exist" Sep 18 16:39:49.455423 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.455378 2564 scope.go:117] "RemoveContainer" containerID="f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe" Sep 18 16:39:49.455734 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:39:49.455707 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe\": container with ID starting with f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe not found: ID does not exist" containerID="f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe" Sep 18 16:39:49.455819 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.455741 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe"} err="failed to get container status \"f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe\": rpc error: code = NotFound desc = could not find container \"f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe\": container with ID starting with f345273e5804c53a5c602cf0a7c025b47a0357098e2ea63aea581688d9712dfe not found: ID does not exist" Sep 18 16:39:49.458327 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.458306 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl"] Sep 18 16:39:49.462035 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.462012 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c9ed-predictor-c4c8dc488-hxjcl"] Sep 18 16:39:49.639278 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:49.639251 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" path="/var/lib/kubelet/pods/c533cc1e-39ac-437b-b37b-4720c41e4f3e/volumes" Sep 18 16:39:52.253558 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:52.253526 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:39:54.442723 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:54.442687 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:39:54.443273 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:39:54.443237 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 18 16:40:04.443674 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:40:04.443620 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 18 16:40:14.443513 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:40:14.443451 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 18 16:40:24.443608 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:40:24.443564 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 18 16:40:34.443678 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:40:34.443641 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:40:35.626985 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:40:35.626956 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:40:35.636447 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:40:35.636413 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:40:35.638242 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:40:35.638219 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:40:35.646815 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:40:35.646795 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:45:35.652683 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:45:35.652657 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:45:35.660564 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:45:35.660541 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:45:35.664766 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:45:35.664747 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:45:35.671409 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:45:35.671391 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:48:29.745759 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.745650 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk"] Sep 18 16:48:29.746336 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.745973 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" containerID="cri-o://3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae" gracePeriod=30 Sep 18 16:48:29.746336 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.745997 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kube-rbac-proxy" containerID="cri-o://0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440" gracePeriod=30 Sep 18 16:48:29.791869 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.791830 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf"] Sep 18 16:48:29.792201 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.792188 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kserve-container" Sep 18 16:48:29.792245 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.792204 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kserve-container" Sep 18 16:48:29.792245 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.792228 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kube-rbac-proxy" Sep 18 16:48:29.792245 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.792234 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kube-rbac-proxy" Sep 18 16:48:29.792342 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.792281 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kube-rbac-proxy" Sep 18 16:48:29.792342 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.792290 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c533cc1e-39ac-437b-b37b-4720c41e4f3e" containerName="kserve-container" Sep 18 16:48:29.797634 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.797575 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:29.800797 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.800746 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b3de6-kube-rbac-proxy-sar-config\"" Sep 18 16:48:29.800797 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.800768 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b3de6-predictor-serving-cert\"" Sep 18 16:48:29.805192 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.805161 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf"] Sep 18 16:48:29.870390 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.870346 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqfnw\" (UniqueName: \"kubernetes.io/projected/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-api-access-pqfnw\") pod \"error-404-isvc-b3de6-predictor-789679b4dd-5cfbf\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:29.870572 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.870420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b3de6-predictor-789679b4dd-5cfbf\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:29.870572 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.870544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-proxy-tls\") pod \"error-404-isvc-b3de6-predictor-789679b4dd-5cfbf\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:29.971545 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.971461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-proxy-tls\") pod \"error-404-isvc-b3de6-predictor-789679b4dd-5cfbf\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:29.971853 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.971608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqfnw\" (UniqueName: \"kubernetes.io/projected/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-api-access-pqfnw\") pod \"error-404-isvc-b3de6-predictor-789679b4dd-5cfbf\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:29.971853 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.971660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b3de6-predictor-789679b4dd-5cfbf\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:29.972390 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.972359 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b3de6-predictor-789679b4dd-5cfbf\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:29.974321 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.974285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-proxy-tls\") pod \"error-404-isvc-b3de6-predictor-789679b4dd-5cfbf\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:29.982851 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:29.982818 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqfnw\" (UniqueName: \"kubernetes.io/projected/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-api-access-pqfnw\") pod \"error-404-isvc-b3de6-predictor-789679b4dd-5cfbf\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:30.110559 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:30.110414 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:30.406740 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:30.406634 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf"] Sep 18 16:48:30.409855 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:48:30.409827 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d20b97_e9ed_4b18_8463_e3dfe8f907a1.slice/crio-69597e0035ba25eda0e2218fe918d11a1494678cca1860f847d63ef5416d2b34 WatchSource:0}: Error finding container 69597e0035ba25eda0e2218fe918d11a1494678cca1860f847d63ef5416d2b34: Status 404 returned error can't find the container with id 69597e0035ba25eda0e2218fe918d11a1494678cca1860f847d63ef5416d2b34 Sep 18 16:48:30.546362 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:30.546326 2564 generic.go:356] "Generic (PLEG): container finished" podID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerID="0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440" exitCode=2 Sep 18 16:48:30.546584 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:30.546397 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" event={"ID":"fb03191d-d415-46ec-9ae0-bd9af3340e6c","Type":"ContainerDied","Data":"0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440"} Sep 18 16:48:30.548417 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:30.548385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" event={"ID":"09d20b97-e9ed-4b18-8463-e3dfe8f907a1","Type":"ContainerStarted","Data":"c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52"} Sep 18 16:48:30.548570 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:30.548447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" event={"ID":"09d20b97-e9ed-4b18-8463-e3dfe8f907a1","Type":"ContainerStarted","Data":"58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9"} Sep 18 16:48:30.548570 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:30.548472 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" event={"ID":"09d20b97-e9ed-4b18-8463-e3dfe8f907a1","Type":"ContainerStarted","Data":"69597e0035ba25eda0e2218fe918d11a1494678cca1860f847d63ef5416d2b34"} Sep 18 16:48:30.548570 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:30.548534 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:30.567340 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:30.567279 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podStartSLOduration=1.567258318 podStartE2EDuration="1.567258318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:48:30.566297443 +0000 UTC m=+1977.085564942" watchObservedRunningTime="2026-09-18 16:48:30.567258318 +0000 UTC m=+1977.086525814" Sep 18 16:48:31.554619 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:31.554573 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:31.556363 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:31.556316 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 18 16:48:32.246255 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:32.246207 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused" Sep 18 16:48:32.252774 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:32.252722 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 18 16:48:32.559296 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:32.559184 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 18 16:48:34.206526 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.206413 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:48:34.311238 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.311194 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb03191d-d415-46ec-9ae0-bd9af3340e6c-proxy-tls\") pod \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " Sep 18 16:48:34.311456 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.311268 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-rbac-proxy-sar-config\") pod \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " Sep 18 16:48:34.311456 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.311326 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8kg2\" (UniqueName: \"kubernetes.io/projected/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-api-access-p8kg2\") pod \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\" (UID: \"fb03191d-d415-46ec-9ae0-bd9af3340e6c\") " Sep 18 16:48:34.311772 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.311742 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb03191d-d415-46ec-9ae0-bd9af3340e6c" (UID: "fb03191d-d415-46ec-9ae0-bd9af3340e6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:48:34.313624 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.313595 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-api-access-p8kg2" (OuterVolumeSpecName: "kube-api-access-p8kg2") pod "fb03191d-d415-46ec-9ae0-bd9af3340e6c" (UID: "fb03191d-d415-46ec-9ae0-bd9af3340e6c"). InnerVolumeSpecName "kube-api-access-p8kg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:48:34.313745 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.313717 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb03191d-d415-46ec-9ae0-bd9af3340e6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb03191d-d415-46ec-9ae0-bd9af3340e6c" (UID: "fb03191d-d415-46ec-9ae0-bd9af3340e6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:48:34.412053 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.412003 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb03191d-d415-46ec-9ae0-bd9af3340e6c-proxy-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:48:34.412053 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.412038 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:48:34.412053 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.412051 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8kg2\" (UniqueName: \"kubernetes.io/projected/fb03191d-d415-46ec-9ae0-bd9af3340e6c-kube-api-access-p8kg2\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:48:34.572750 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.572647 2564 generic.go:356] "Generic (PLEG): container finished" podID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerID="3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae" exitCode=0 Sep 18 16:48:34.572750 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.572702 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" Sep 18 16:48:34.572750 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.572727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" event={"ID":"fb03191d-d415-46ec-9ae0-bd9af3340e6c","Type":"ContainerDied","Data":"3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae"} Sep 18 16:48:34.573036 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.572802 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk" event={"ID":"fb03191d-d415-46ec-9ae0-bd9af3340e6c","Type":"ContainerDied","Data":"6650c0d28554c01ccd0109e600d2624227e9138889b3f6d3b4057d61e613bfd1"} Sep 18 16:48:34.573036 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.572834 2564 scope.go:117] "RemoveContainer" containerID="0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440" Sep 18 16:48:34.582716 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.582691 2564 scope.go:117] "RemoveContainer" containerID="3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae" Sep 18 16:48:34.591379 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.591359 2564 scope.go:117] "RemoveContainer" containerID="0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440" Sep 18 16:48:34.591761 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:48:34.591740 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440\": container with ID starting with 0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440 not found: ID does not exist" containerID="0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440" Sep 18 16:48:34.591818 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.591771 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440"} err="failed to get container status \"0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440\": rpc error: code = NotFound desc = could not find container \"0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440\": container with ID starting with 0a2da9fb8b747179bd1eb4218adc60c6e2e7bc1b571f6eb93b2cf8e075124440 not found: ID does not exist" Sep 18 16:48:34.591818 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.591791 2564 scope.go:117] "RemoveContainer" containerID="3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae" Sep 18 16:48:34.592030 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:48:34.592016 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae\": container with ID starting with 3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae not found: ID does not exist" containerID="3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae" Sep 18 16:48:34.592066 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.592033 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae"} err="failed to get container status \"3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae\": rpc error: code = NotFound desc = could not find container \"3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae\": container with ID starting with 3c5161beb9ea3eb34967ad01ee43ec76259153e5159064c770f7cb45ad5725ae not found: ID does not exist" Sep 18 16:48:34.596568 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.596519 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk"] Sep 18 16:48:34.600345 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:34.600314 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de495-predictor-7fb5ffb98b-ztgqk"] Sep 18 16:48:35.640573 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:35.640536 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" path="/var/lib/kubelet/pods/fb03191d-d415-46ec-9ae0-bd9af3340e6c/volumes" Sep 18 16:48:37.564533 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:37.564499 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:48:37.565038 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:37.565009 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 18 16:48:47.565042 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:47.564996 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 18 16:48:57.565549 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:48:57.565474 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 18 16:49:07.565808 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:07.565754 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 18 16:49:10.976561 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:10.976069 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb"] Sep 18 16:49:10.976561 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:10.976466 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" containerID="cri-o://dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06" gracePeriod=30 Sep 18 16:49:10.977127 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:10.977017 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kube-rbac-proxy" containerID="cri-o://163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37" gracePeriod=30 Sep 18 16:49:11.194452 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.194407 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5"] Sep 18 16:49:11.194902 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.194881 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" Sep 18 16:49:11.194984 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.194904 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" Sep 18 16:49:11.194984 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.194920 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kube-rbac-proxy" Sep 18 16:49:11.194984 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.194925 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kube-rbac-proxy" Sep 18 16:49:11.195078 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.194985 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kube-rbac-proxy" Sep 18 16:49:11.195078 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.194993 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb03191d-d415-46ec-9ae0-bd9af3340e6c" containerName="kserve-container" Sep 18 16:49:11.198177 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.198153 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:11.200892 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.200863 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a2696-predictor-serving-cert\"" Sep 18 16:49:11.201016 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.200863 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a2696-kube-rbac-proxy-sar-config\"" Sep 18 16:49:11.212918 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.212879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5"] Sep 18 16:49:11.248255 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.248159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a2696-predictor-548b77d64-8zrh5\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:11.248426 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.248296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6qq\" (UniqueName: \"kubernetes.io/projected/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-api-access-5r6qq\") pod \"error-404-isvc-a2696-predictor-548b77d64-8zrh5\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:11.248426 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.248341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2686ece5-1634-4763-9774-a5eb35d7cbb4-proxy-tls\") pod \"error-404-isvc-a2696-predictor-548b77d64-8zrh5\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:11.348859 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.348809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6qq\" (UniqueName: \"kubernetes.io/projected/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-api-access-5r6qq\") pod \"error-404-isvc-a2696-predictor-548b77d64-8zrh5\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:11.348859 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.348860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2686ece5-1634-4763-9774-a5eb35d7cbb4-proxy-tls\") pod \"error-404-isvc-a2696-predictor-548b77d64-8zrh5\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:11.349155 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.348898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a2696-predictor-548b77d64-8zrh5\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:11.349155 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:49:11.349034 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a2696-predictor-serving-cert: secret "error-404-isvc-a2696-predictor-serving-cert" not found Sep 18 16:49:11.349155 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:49:11.349142 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2686ece5-1634-4763-9774-a5eb35d7cbb4-proxy-tls podName:2686ece5-1634-4763-9774-a5eb35d7cbb4 nodeName:}" failed. No retries permitted until 2026-09-18 16:49:11.849123585 +0000 UTC m=+2018.368391062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2686ece5-1634-4763-9774-a5eb35d7cbb4-proxy-tls") pod "error-404-isvc-a2696-predictor-548b77d64-8zrh5" (UID: "2686ece5-1634-4763-9774-a5eb35d7cbb4") : secret "error-404-isvc-a2696-predictor-serving-cert" not found Sep 18 16:49:11.349618 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.349600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a2696-predictor-548b77d64-8zrh5\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:11.358815 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.358772 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6qq\" (UniqueName: \"kubernetes.io/projected/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-api-access-5r6qq\") pod \"error-404-isvc-a2696-predictor-548b77d64-8zrh5\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:11.741581 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.741539 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1518096-00eb-41fb-96e4-d54adf968795" containerID="163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37" exitCode=2 Sep 18 16:49:11.741772 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.741615 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" event={"ID":"d1518096-00eb-41fb-96e4-d54adf968795","Type":"ContainerDied","Data":"163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37"} Sep 18 16:49:11.854339 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.854282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2686ece5-1634-4763-9774-a5eb35d7cbb4-proxy-tls\") pod \"error-404-isvc-a2696-predictor-548b77d64-8zrh5\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:11.856945 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:11.856903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2686ece5-1634-4763-9774-a5eb35d7cbb4-proxy-tls\") pod \"error-404-isvc-a2696-predictor-548b77d64-8zrh5\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:12.109075 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:12.108962 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:12.387107 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:12.387012 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5"] Sep 18 16:49:12.390348 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:49:12.390311 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2686ece5_1634_4763_9774_a5eb35d7cbb4.slice/crio-3f028494f2b980d5fd63c6df563df1cf461d43539dce7ae85e9c0aa0a0311b00 WatchSource:0}: Error finding container 3f028494f2b980d5fd63c6df563df1cf461d43539dce7ae85e9c0aa0a0311b00: Status 404 returned error can't find the container with id 3f028494f2b980d5fd63c6df563df1cf461d43539dce7ae85e9c0aa0a0311b00 Sep 18 16:49:12.748803 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:12.748765 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" event={"ID":"2686ece5-1634-4763-9774-a5eb35d7cbb4","Type":"ContainerStarted","Data":"e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48"} Sep 18 16:49:12.749027 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:12.748825 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" event={"ID":"2686ece5-1634-4763-9774-a5eb35d7cbb4","Type":"ContainerStarted","Data":"baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054"} Sep 18 16:49:12.749027 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:12.748850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" event={"ID":"2686ece5-1634-4763-9774-a5eb35d7cbb4","Type":"ContainerStarted","Data":"3f028494f2b980d5fd63c6df563df1cf461d43539dce7ae85e9c0aa0a0311b00"} Sep 18 16:49:12.749027 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:12.748880 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:12.767776 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:12.767716 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" podStartSLOduration=1.7676977809999999 podStartE2EDuration="1.767697781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:49:12.766355272 +0000 UTC m=+2019.285622776" watchObservedRunningTime="2026-09-18 16:49:12.767697781 +0000 UTC m=+2019.286965280" Sep 18 16:49:13.753971 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:13.753940 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:13.755175 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:13.755139 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 16:49:14.438891 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:14.438836 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused" Sep 18 16:49:14.443300 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:14.443265 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 18 16:49:14.760219 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:14.760085 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 16:49:15.538114 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.538071 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:49:15.590285 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.590241 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1518096-00eb-41fb-96e4-d54adf968795-proxy-tls\") pod \"d1518096-00eb-41fb-96e4-d54adf968795\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " Sep 18 16:49:15.590537 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.590362 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1518096-00eb-41fb-96e4-d54adf968795-kube-rbac-proxy-sar-config\") pod \"d1518096-00eb-41fb-96e4-d54adf968795\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " Sep 18 16:49:15.590537 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.590387 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgvb5\" (UniqueName: \"kubernetes.io/projected/d1518096-00eb-41fb-96e4-d54adf968795-kube-api-access-fgvb5\") pod \"d1518096-00eb-41fb-96e4-d54adf968795\" (UID: \"d1518096-00eb-41fb-96e4-d54adf968795\") " Sep 18 16:49:15.590792 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.590763 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1518096-00eb-41fb-96e4-d54adf968795-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1518096-00eb-41fb-96e4-d54adf968795" (UID: "d1518096-00eb-41fb-96e4-d54adf968795"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:49:15.592757 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.592713 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1518096-00eb-41fb-96e4-d54adf968795-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1518096-00eb-41fb-96e4-d54adf968795" (UID: "d1518096-00eb-41fb-96e4-d54adf968795"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:49:15.592757 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.592742 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1518096-00eb-41fb-96e4-d54adf968795-kube-api-access-fgvb5" (OuterVolumeSpecName: "kube-api-access-fgvb5") pod "d1518096-00eb-41fb-96e4-d54adf968795" (UID: "d1518096-00eb-41fb-96e4-d54adf968795"). InnerVolumeSpecName "kube-api-access-fgvb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:49:15.691939 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.691888 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1518096-00eb-41fb-96e4-d54adf968795-proxy-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:49:15.691939 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.691932 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1518096-00eb-41fb-96e4-d54adf968795-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:49:15.691939 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.691943 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgvb5\" (UniqueName: \"kubernetes.io/projected/d1518096-00eb-41fb-96e4-d54adf968795-kube-api-access-fgvb5\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:49:15.770627 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.770584 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1518096-00eb-41fb-96e4-d54adf968795" containerID="dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06" exitCode=0 Sep 18 16:49:15.771063 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.770653 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" Sep 18 16:49:15.771063 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.770658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" event={"ID":"d1518096-00eb-41fb-96e4-d54adf968795","Type":"ContainerDied","Data":"dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06"} Sep 18 16:49:15.771063 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.770738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb" event={"ID":"d1518096-00eb-41fb-96e4-d54adf968795","Type":"ContainerDied","Data":"1d13e489e5c3e1dc04db6aa0aca70d1c7f3d37c3efa622ddaf372cbf7ec5e9a8"} Sep 18 16:49:15.771063 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.770776 2564 scope.go:117] "RemoveContainer" containerID="163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37" Sep 18 16:49:15.779630 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.779610 2564 scope.go:117] "RemoveContainer" containerID="dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06" Sep 18 16:49:15.789428 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.789397 2564 scope.go:117] "RemoveContainer" containerID="163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37" Sep 18 16:49:15.789958 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:49:15.789933 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37\": container with ID starting with 163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37 not found: ID does not exist" containerID="163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37" Sep 18 16:49:15.790087 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.789968 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37"} err="failed to get container status \"163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37\": rpc error: code = NotFound desc = could not find container \"163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37\": container with ID starting with 163856901ad80a26dbbf44000122e0f1ba272de3aa6b2476b5316138a848dd37 not found: ID does not exist" Sep 18 16:49:15.790087 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.789993 2564 scope.go:117] "RemoveContainer" containerID="dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06" Sep 18 16:49:15.790353 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:49:15.790329 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06\": container with ID starting with dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06 not found: ID does not exist" containerID="dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06" Sep 18 16:49:15.790414 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.790358 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06"} err="failed to get container status \"dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06\": rpc error: code = NotFound desc = could not find container \"dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06\": container with ID starting with dff8a9bf8173d39dd2b1afdcd918babcdc39c0ed435e1b60a81ac9487b86cc06 not found: ID does not exist" Sep 18 16:49:15.790734 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.790705 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb"] Sep 18 16:49:15.799033 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:15.798997 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-630d7-predictor-7bc45d9448-6gblb"] Sep 18 16:49:17.565447 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:17.565402 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 18 16:49:17.640820 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:17.640777 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1518096-00eb-41fb-96e4-d54adf968795" path="/var/lib/kubelet/pods/d1518096-00eb-41fb-96e4-d54adf968795/volumes" Sep 18 16:49:19.764234 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:19.764203 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:49:19.764743 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:19.764714 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 16:49:27.566604 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:27.566563 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:49:29.765351 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:29.765302 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 16:49:39.765333 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:39.765230 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 16:49:49.764763 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:49.764713 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 16:49:50.929979 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:50.929933 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf"] Sep 18 16:49:50.930479 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:50.930301 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" containerID="cri-o://58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9" gracePeriod=30 Sep 18 16:49:50.930479 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:50.930410 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kube-rbac-proxy" containerID="cri-o://c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52" gracePeriod=30 Sep 18 16:49:51.119917 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.119874 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t"] Sep 18 16:49:51.120279 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.120266 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" Sep 18 16:49:51.120328 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.120282 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" Sep 18 16:49:51.120328 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.120310 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kube-rbac-proxy" Sep 18 16:49:51.120328 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.120316 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kube-rbac-proxy" Sep 18 16:49:51.120418 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.120402 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kserve-container" Sep 18 16:49:51.120418 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.120411 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1518096-00eb-41fb-96e4-d54adf968795" containerName="kube-rbac-proxy" Sep 18 16:49:51.123717 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.123695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.126150 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.126118 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bb388-kube-rbac-proxy-sar-config\"" Sep 18 16:49:51.126342 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.126117 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bb388-predictor-serving-cert\"" Sep 18 16:49:51.133646 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.133609 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t"] Sep 18 16:49:51.201706 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.201666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-proxy-tls\") pod \"error-404-isvc-bb388-predictor-6d44f64874-nwx7t\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.201706 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.201710 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zll9\" (UniqueName: \"kubernetes.io/projected/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-api-access-6zll9\") pod \"error-404-isvc-bb388-predictor-6d44f64874-nwx7t\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.201937 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.201751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bb388-predictor-6d44f64874-nwx7t\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.302241 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.302191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bb388-predictor-6d44f64874-nwx7t\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.302457 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.302284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-proxy-tls\") pod \"error-404-isvc-bb388-predictor-6d44f64874-nwx7t\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.302457 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.302306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zll9\" (UniqueName: \"kubernetes.io/projected/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-api-access-6zll9\") pod \"error-404-isvc-bb388-predictor-6d44f64874-nwx7t\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.302701 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:49:51.302457 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-bb388-predictor-serving-cert: secret "error-404-isvc-bb388-predictor-serving-cert" not found Sep 18 16:49:51.302701 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:49:51.302591 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-proxy-tls podName:1fdb4863-f39d-41db-bf3c-1b04a45ac05b nodeName:}" failed. No retries permitted until 2026-09-18 16:49:51.802567765 +0000 UTC m=+2058.321835259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-proxy-tls") pod "error-404-isvc-bb388-predictor-6d44f64874-nwx7t" (UID: "1fdb4863-f39d-41db-bf3c-1b04a45ac05b") : secret "error-404-isvc-bb388-predictor-serving-cert" not found Sep 18 16:49:51.302946 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.302925 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bb388-predictor-6d44f64874-nwx7t\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.314199 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.314173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zll9\" (UniqueName: \"kubernetes.io/projected/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-api-access-6zll9\") pod \"error-404-isvc-bb388-predictor-6d44f64874-nwx7t\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.805615 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.805569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-proxy-tls\") pod \"error-404-isvc-bb388-predictor-6d44f64874-nwx7t\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.808098 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.808072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-proxy-tls\") pod \"error-404-isvc-bb388-predictor-6d44f64874-nwx7t\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:51.936456 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.936414 2564 generic.go:356] "Generic (PLEG): container finished" podID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerID="c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52" exitCode=2 Sep 18 16:49:51.936881 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:51.936497 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" event={"ID":"09d20b97-e9ed-4b18-8463-e3dfe8f907a1","Type":"ContainerDied","Data":"c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52"} Sep 18 16:49:52.036122 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:52.036074 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:52.314139 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:52.314077 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t"] Sep 18 16:49:52.317433 ip-10-0-143-79 kubenswrapper[2564]: W0918 16:49:52.317394 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdb4863_f39d_41db_bf3c_1b04a45ac05b.slice/crio-4652113032c42bedb9816324863c6e0b472685a9dc397a5829ada8efe1986bcb WatchSource:0}: Error finding container 4652113032c42bedb9816324863c6e0b472685a9dc397a5829ada8efe1986bcb: Status 404 returned error can't find the container with id 4652113032c42bedb9816324863c6e0b472685a9dc397a5829ada8efe1986bcb Sep 18 16:49:52.559804 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:52.559740 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused" Sep 18 16:49:52.942739 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:52.942694 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" event={"ID":"1fdb4863-f39d-41db-bf3c-1b04a45ac05b","Type":"ContainerStarted","Data":"28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176"} Sep 18 16:49:52.943246 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:52.942768 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" event={"ID":"1fdb4863-f39d-41db-bf3c-1b04a45ac05b","Type":"ContainerStarted","Data":"23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110"} Sep 18 16:49:52.943246 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:52.942930 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:52.943246 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:52.942974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" event={"ID":"1fdb4863-f39d-41db-bf3c-1b04a45ac05b","Type":"ContainerStarted","Data":"4652113032c42bedb9816324863c6e0b472685a9dc397a5829ada8efe1986bcb"} Sep 18 16:49:52.964266 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:52.964196 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podStartSLOduration=1.964176636 podStartE2EDuration="1.964176636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:49:52.961346424 +0000 UTC m=+2059.480613933" watchObservedRunningTime="2026-09-18 16:49:52.964176636 +0000 UTC m=+2059.483444136" Sep 18 16:49:53.947475 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:53.947440 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:53.949061 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:53.949022 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 16:49:54.952570 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:54.952523 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 16:49:55.388503 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.388427 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:49:55.434389 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.434349 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-proxy-tls\") pod \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " Sep 18 16:49:55.434633 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.434400 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqfnw\" (UniqueName: \"kubernetes.io/projected/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-api-access-pqfnw\") pod \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " Sep 18 16:49:55.434633 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.434441 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-rbac-proxy-sar-config\") pod \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\" (UID: \"09d20b97-e9ed-4b18-8463-e3dfe8f907a1\") " Sep 18 16:49:55.434925 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.434893 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09d20b97-e9ed-4b18-8463-e3dfe8f907a1" (UID: "09d20b97-e9ed-4b18-8463-e3dfe8f907a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:49:55.436724 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.436691 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-api-access-pqfnw" (OuterVolumeSpecName: "kube-api-access-pqfnw") pod "09d20b97-e9ed-4b18-8463-e3dfe8f907a1" (UID: "09d20b97-e9ed-4b18-8463-e3dfe8f907a1"). InnerVolumeSpecName "kube-api-access-pqfnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:49:55.436830 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.436704 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09d20b97-e9ed-4b18-8463-e3dfe8f907a1" (UID: "09d20b97-e9ed-4b18-8463-e3dfe8f907a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:49:55.535462 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.535367 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-proxy-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:49:55.535462 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.535401 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqfnw\" (UniqueName: \"kubernetes.io/projected/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-api-access-pqfnw\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:49:55.535462 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.535411 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d20b97-e9ed-4b18-8463-e3dfe8f907a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:49:55.671696 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:49:55.671649 2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d20b97_e9ed_4b18_8463_e3dfe8f907a1.slice\": RecentStats: unable to find data in memory cache]" Sep 18 16:49:55.959090 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.959050 2564 generic.go:356] "Generic (PLEG): container finished" podID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerID="58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9" exitCode=0 Sep 18 16:49:55.959638 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.959111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" event={"ID":"09d20b97-e9ed-4b18-8463-e3dfe8f907a1","Type":"ContainerDied","Data":"58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9"} Sep 18 16:49:55.959638 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.959136 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" Sep 18 16:49:55.959638 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.959190 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf" event={"ID":"09d20b97-e9ed-4b18-8463-e3dfe8f907a1","Type":"ContainerDied","Data":"69597e0035ba25eda0e2218fe918d11a1494678cca1860f847d63ef5416d2b34"} Sep 18 16:49:55.959638 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.959218 2564 scope.go:117] "RemoveContainer" containerID="c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52" Sep 18 16:49:55.968382 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.968365 2564 scope.go:117] "RemoveContainer" containerID="58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9" Sep 18 16:49:55.977341 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.977314 2564 scope.go:117] "RemoveContainer" containerID="c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52" Sep 18 16:49:55.977680 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.977659 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf"] Sep 18 16:49:55.977781 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:49:55.977764 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52\": container with ID starting with c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52 not found: ID does not exist" containerID="c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52" Sep 18 16:49:55.977825 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.977791 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52"} err="failed to get container status \"c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52\": rpc error: code = NotFound desc = could not find container \"c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52\": container with ID starting with c66260f0b5d414501020c45f990b34ec9768f0d802475426c7cea856b87ecf52 not found: ID does not exist" Sep 18 16:49:55.977825 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.977810 2564 scope.go:117] "RemoveContainer" containerID="58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9" Sep 18 16:49:55.978102 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:49:55.978086 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9\": container with ID starting with 58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9 not found: ID does not exist" containerID="58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9" Sep 18 16:49:55.978147 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.978106 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9"} err="failed to get container status \"58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9\": rpc error: code = NotFound desc = could not find container \"58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9\": container with ID starting with 58103b00d3478676bfd10ff5ca70cc7e4098dd1a718531f3160383771d64aed9 not found: ID does not exist" Sep 18 16:49:55.982167 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:55.982133 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b3de6-predictor-789679b4dd-5cfbf"] Sep 18 16:49:57.640662 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:57.640623 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" path="/var/lib/kubelet/pods/09d20b97-e9ed-4b18-8463-e3dfe8f907a1/volumes" Sep 18 16:49:59.764880 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:59.764839 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 16:49:59.958662 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:59.958623 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:49:59.959239 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:49:59.959211 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 16:50:09.771753 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:50:09.771706 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 16:50:09.959838 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:50:09.959763 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 16:50:19.960069 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:50:19.960019 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 16:50:29.959561 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:50:29.959479 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 16:50:35.679997 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:50:35.679960 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:50:35.688408 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:50:35.688377 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:50:35.690875 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:50:35.690849 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:50:35.699458 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:50:35.699425 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:50:39.960196 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:50:39.960135 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 16:50:49.961099 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:50:49.961059 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:55:35.705176 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:55:35.705148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:55:35.713008 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:55:35.712987 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:55:35.717289 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:55:35.717270 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 16:55:35.724844 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:55:35.724829 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 16:59:16.229734 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:16.229684 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t"] Sep 18 16:59:16.230360 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:16.230040 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" containerID="cri-o://23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110" gracePeriod=30 Sep 18 16:59:16.230360 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:16.230057 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kube-rbac-proxy" containerID="cri-o://28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176" gracePeriod=30 Sep 18 16:59:17.327026 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:17.326977 2564 generic.go:356] "Generic (PLEG): container finished" podID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerID="28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176" exitCode=2 Sep 18 16:59:17.327443 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:17.327055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" event={"ID":"1fdb4863-f39d-41db-bf3c-1b04a45ac05b","Type":"ContainerDied","Data":"28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176"} Sep 18 16:59:19.953161 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:19.953099 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Sep 18 16:59:19.959715 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:19.959662 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 16:59:20.798638 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:20.798602 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:59:20.942225 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:20.942185 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zll9\" (UniqueName: \"kubernetes.io/projected/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-api-access-6zll9\") pod \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " Sep 18 16:59:20.942411 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:20.942234 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-rbac-proxy-sar-config\") pod \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " Sep 18 16:59:20.942411 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:20.942287 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-proxy-tls\") pod \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\" (UID: \"1fdb4863-f39d-41db-bf3c-1b04a45ac05b\") " Sep 18 16:59:20.942689 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:20.942663 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1fdb4863-f39d-41db-bf3c-1b04a45ac05b" (UID: "1fdb4863-f39d-41db-bf3c-1b04a45ac05b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:59:20.944643 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:20.944610 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1fdb4863-f39d-41db-bf3c-1b04a45ac05b" (UID: "1fdb4863-f39d-41db-bf3c-1b04a45ac05b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:59:20.944726 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:20.944703 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-api-access-6zll9" (OuterVolumeSpecName: "kube-api-access-6zll9") pod "1fdb4863-f39d-41db-bf3c-1b04a45ac05b" (UID: "1fdb4863-f39d-41db-bf3c-1b04a45ac05b"). InnerVolumeSpecName "kube-api-access-6zll9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:59:21.043600 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.043553 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zll9\" (UniqueName: \"kubernetes.io/projected/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-api-access-6zll9\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:59:21.043600 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.043591 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:59:21.043600 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.043604 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdb4863-f39d-41db-bf3c-1b04a45ac05b-proxy-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 16:59:21.345419 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.345314 2564 generic.go:356] "Generic (PLEG): container finished" podID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerID="23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110" exitCode=0 Sep 18 16:59:21.345419 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.345365 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" Sep 18 16:59:21.345419 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.345399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" event={"ID":"1fdb4863-f39d-41db-bf3c-1b04a45ac05b","Type":"ContainerDied","Data":"23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110"} Sep 18 16:59:21.345703 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.345461 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t" event={"ID":"1fdb4863-f39d-41db-bf3c-1b04a45ac05b","Type":"ContainerDied","Data":"4652113032c42bedb9816324863c6e0b472685a9dc397a5829ada8efe1986bcb"} Sep 18 16:59:21.345703 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.345505 2564 scope.go:117] "RemoveContainer" containerID="28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176" Sep 18 16:59:21.354883 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.354862 2564 scope.go:117] "RemoveContainer" containerID="23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110" Sep 18 16:59:21.363833 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.363809 2564 scope.go:117] "RemoveContainer" containerID="28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176" Sep 18 16:59:21.364205 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:59:21.364177 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176\": container with ID starting with 28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176 not found: ID does not exist" containerID="28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176" Sep 18 16:59:21.364328 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.364211 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176"} err="failed to get container status \"28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176\": rpc error: code = NotFound desc = could not find container \"28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176\": container with ID starting with 28eed1f53c15d9ca1378a111c14a7ec5c46216b2a9bad824f139bfc4b4cba176 not found: ID does not exist" Sep 18 16:59:21.364328 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.364232 2564 scope.go:117] "RemoveContainer" containerID="23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110" Sep 18 16:59:21.364537 ip-10-0-143-79 kubenswrapper[2564]: E0918 16:59:21.364521 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110\": container with ID starting with 23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110 not found: ID does not exist" containerID="23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110" Sep 18 16:59:21.364595 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.364541 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110"} err="failed to get container status \"23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110\": rpc error: code = NotFound desc = could not find container \"23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110\": container with ID starting with 23a73b6cb81fa797ceeccafa86de592567275b70605c259ad49f04bde02b6110 not found: ID does not exist" Sep 18 16:59:21.367626 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.367589 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t"] Sep 18 16:59:21.371898 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.371855 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb388-predictor-6d44f64874-nwx7t"] Sep 18 16:59:21.640162 ip-10-0-143-79 kubenswrapper[2564]: I0918 16:59:21.640074 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" path="/var/lib/kubelet/pods/1fdb4863-f39d-41db-bf3c-1b04a45ac05b/volumes" Sep 18 17:00:35.730242 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:00:35.730117 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 17:00:35.738967 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:00:35.738934 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 17:00:35.743349 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:00:35.743322 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 17:00:35.751349 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:00:35.751321 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 17:05:35.755411 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:05:35.755318 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 17:05:35.764354 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:05:35.764329 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 17:05:35.770011 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:05:35.769983 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 17:05:35.778342 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:05:35.778320 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 17:06:41.294906 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:41.294828 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5"] Sep 18 17:06:41.295421 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:41.295076 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" containerID="cri-o://baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054" gracePeriod=30 Sep 18 17:06:41.295421 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:41.295112 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kube-rbac-proxy" containerID="cri-o://e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48" gracePeriod=30 Sep 18 17:06:42.211406 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:42.211372 2564 generic.go:356] "Generic (PLEG): container finished" podID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerID="e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48" exitCode=2 Sep 18 17:06:42.211591 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:42.211448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" event={"ID":"2686ece5-1634-4763-9774-a5eb35d7cbb4","Type":"ContainerDied","Data":"e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48"} Sep 18 17:06:43.931870 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:43.931846 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 17:06:44.036876 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.036823 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2686ece5-1634-4763-9774-a5eb35d7cbb4-proxy-tls\") pod \"2686ece5-1634-4763-9774-a5eb35d7cbb4\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " Sep 18 17:06:44.036976 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.036887 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r6qq\" (UniqueName: \"kubernetes.io/projected/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-api-access-5r6qq\") pod \"2686ece5-1634-4763-9774-a5eb35d7cbb4\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " Sep 18 17:06:44.036976 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.036911 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-rbac-proxy-sar-config\") pod \"2686ece5-1634-4763-9774-a5eb35d7cbb4\" (UID: \"2686ece5-1634-4763-9774-a5eb35d7cbb4\") " Sep 18 17:06:44.037274 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.037243 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2686ece5-1634-4763-9774-a5eb35d7cbb4" (UID: "2686ece5-1634-4763-9774-a5eb35d7cbb4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:06:44.038767 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.038734 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2686ece5-1634-4763-9774-a5eb35d7cbb4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2686ece5-1634-4763-9774-a5eb35d7cbb4" (UID: "2686ece5-1634-4763-9774-a5eb35d7cbb4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:06:44.038883 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.038862 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-api-access-5r6qq" (OuterVolumeSpecName: "kube-api-access-5r6qq") pod "2686ece5-1634-4763-9774-a5eb35d7cbb4" (UID: "2686ece5-1634-4763-9774-a5eb35d7cbb4"). InnerVolumeSpecName "kube-api-access-5r6qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:06:44.138233 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.138208 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r6qq\" (UniqueName: \"kubernetes.io/projected/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-api-access-5r6qq\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 17:06:44.138233 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.138229 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2686ece5-1634-4763-9774-a5eb35d7cbb4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 17:06:44.138347 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.138239 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2686ece5-1634-4763-9774-a5eb35d7cbb4-proxy-tls\") on node \"ip-10-0-143-79.ec2.internal\" DevicePath \"\"" Sep 18 17:06:44.220274 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.220246 2564 generic.go:356] "Generic (PLEG): container finished" podID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerID="baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054" exitCode=0 Sep 18 17:06:44.220362 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.220293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" event={"ID":"2686ece5-1634-4763-9774-a5eb35d7cbb4","Type":"ContainerDied","Data":"baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054"} Sep 18 17:06:44.220362 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.220303 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" Sep 18 17:06:44.220362 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.220356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5" event={"ID":"2686ece5-1634-4763-9774-a5eb35d7cbb4","Type":"ContainerDied","Data":"3f028494f2b980d5fd63c6df563df1cf461d43539dce7ae85e9c0aa0a0311b00"} Sep 18 17:06:44.220463 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.220393 2564 scope.go:117] "RemoveContainer" containerID="e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48" Sep 18 17:06:44.229079 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.229064 2564 scope.go:117] "RemoveContainer" containerID="baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054" Sep 18 17:06:44.236948 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.236925 2564 scope.go:117] "RemoveContainer" containerID="e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48" Sep 18 17:06:44.237220 ip-10-0-143-79 kubenswrapper[2564]: E0918 17:06:44.237199 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48\": container with ID starting with e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48 not found: ID does not exist" containerID="e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48" Sep 18 17:06:44.237274 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.237229 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48"} err="failed to get container status \"e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48\": rpc error: code = NotFound desc = could not find container \"e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48\": container with ID starting with e939d2d174fe593fa9432d0618d7285e840e9a23f5fb3bb8875c4af6c8994e48 not found: ID does not exist" Sep 18 17:06:44.237274 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.237247 2564 scope.go:117] "RemoveContainer" containerID="baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054" Sep 18 17:06:44.237518 ip-10-0-143-79 kubenswrapper[2564]: E0918 17:06:44.237478 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054\": container with ID starting with baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054 not found: ID does not exist" containerID="baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054" Sep 18 17:06:44.237584 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.237530 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054"} err="failed to get container status \"baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054\": rpc error: code = NotFound desc = could not find container \"baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054\": container with ID starting with baffd88c940439546f1ac3890a8b908823ea26cb1f22b0e2a4c19202bf16a054 not found: ID does not exist" Sep 18 17:06:44.241349 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.241326 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5"] Sep 18 17:06:44.243322 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:44.243299 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a2696-predictor-548b77d64-8zrh5"] Sep 18 17:06:45.638996 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:06:45.638961 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" path="/var/lib/kubelet/pods/2686ece5-1634-4763-9774-a5eb35d7cbb4/volumes" Sep 18 17:07:15.354262 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:15.354229 2564 ???:1] "http: TLS handshake error from 10.0.128.224:60354: EOF" Sep 18 17:07:15.356814 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:15.356794 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-msncf_458d51a9-e012-413e-8dd6-8354a56b8abc/global-pull-secret-syncer/0.log" Sep 18 17:07:15.520915 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:15.520884 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xplqn_17a00ce7-e290-4cc3-8f47-4d9c9d0a1fd4/konnectivity-agent/0.log" Sep 18 17:07:15.654119 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:15.654045 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-79.ec2.internal_6a6af8d9bebf834f6d415238095457ce/haproxy/0.log" Sep 18 17:07:19.346887 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:19.346851 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-4blfd_9c9fda39-70e8-4d3b-9e16-d767847f72e0/cluster-monitoring-operator/0.log" Sep 18 17:07:19.615296 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:19.615219 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4f8tj_6e6305e8-22de-493f-a8f7-b02a6b32e066/node-exporter/0.log" Sep 18 17:07:19.649017 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:19.648986 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4f8tj_6e6305e8-22de-493f-a8f7-b02a6b32e066/kube-rbac-proxy/0.log" Sep 18 17:07:19.700347 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:19.700322 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4f8tj_6e6305e8-22de-493f-a8f7-b02a6b32e066/init-textfile/0.log" Sep 18 17:07:20.355091 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:20.355059 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b5b95d858-mnwjd_14f57712-65b1-4721-aff4-bbbe75c87fa9/telemeter-client/0.log" Sep 18 17:07:20.379433 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:20.379402 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b5b95d858-mnwjd_14f57712-65b1-4721-aff4-bbbe75c87fa9/reload/0.log" Sep 18 17:07:20.401521 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:20.401471 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b5b95d858-mnwjd_14f57712-65b1-4721-aff4-bbbe75c87fa9/kube-rbac-proxy/0.log" Sep 18 17:07:21.843882 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:21.843855 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-ff5dx_584c5c9c-2e21-4c1f-a309-67e12d7d592c/networking-console-plugin/0.log" Sep 18 17:07:22.452128 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:22.452100 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/1.log" Sep 18 17:07:22.456592 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:22.456572 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mbpfm_d14e4595-8b55-4f60-a557-aa750c699157/console-operator/2.log" Sep 18 17:07:23.479993 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.479952 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-c5s8g_52b086e1-8d64-4d18-93c6-f411c3cb7b87/volume-data-source-validator/0.log" Sep 18 17:07:23.929217 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929116 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf"] Sep 18 17:07:23.929430 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929416 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kube-rbac-proxy" Sep 18 17:07:23.929480 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929430 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kube-rbac-proxy" Sep 18 17:07:23.929480 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929447 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" Sep 18 17:07:23.929480 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929453 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" Sep 18 17:07:23.929480 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929462 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" Sep 18 17:07:23.929480 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929467 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" Sep 18 17:07:23.929480 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929474 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kube-rbac-proxy" Sep 18 17:07:23.929480 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929480 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kube-rbac-proxy" Sep 18 17:07:23.929480 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929504 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kube-rbac-proxy" Sep 18 17:07:23.929745 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929510 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kube-rbac-proxy" Sep 18 17:07:23.929745 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929522 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" Sep 18 17:07:23.929745 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929527 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" Sep 18 17:07:23.929745 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929575 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kserve-container" Sep 18 17:07:23.929745 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929583 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kserve-container" Sep 18 17:07:23.929745 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929590 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kube-rbac-proxy" Sep 18 17:07:23.929745 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929596 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2686ece5-1634-4763-9774-a5eb35d7cbb4" containerName="kube-rbac-proxy" Sep 18 17:07:23.929745 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929603 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fdb4863-f39d-41db-bf3c-1b04a45ac05b" containerName="kserve-container" Sep 18 17:07:23.929745 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.929608 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d20b97-e9ed-4b18-8463-e3dfe8f907a1" containerName="kube-rbac-proxy" Sep 18 17:07:23.932520 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.932478 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:23.935369 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.935344 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpw2m\"/\"kube-root-ca.crt\"" Sep 18 17:07:23.936126 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.936102 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zpw2m\"/\"default-dockercfg-2hq6t\"" Sep 18 17:07:23.936239 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.936130 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpw2m\"/\"openshift-service-ca.crt\"" Sep 18 17:07:23.946249 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:23.946228 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf"] Sep 18 17:07:24.069906 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.069882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-proc\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.070009 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.069912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfzl9\" (UniqueName: \"kubernetes.io/projected/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-kube-api-access-vfzl9\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.070009 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.069942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-podres\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.070092 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.070011 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-sys\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.070092 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.070068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-lib-modules\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.171248 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.171209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-sys\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.171353 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.171260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-lib-modules\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.171353 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.171313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-proc\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.171353 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.171336 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-sys\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.171457 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.171407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-proc\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.171457 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.171337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfzl9\" (UniqueName: \"kubernetes.io/projected/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-kube-api-access-vfzl9\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.171457 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.171451 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-lib-modules\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.171596 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.171479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-podres\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.171596 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.171592 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-podres\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.180080 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.180022 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfzl9\" (UniqueName: \"kubernetes.io/projected/d7dcfc21-05ff-4d7f-823f-8b100b78e0b7-kube-api-access-vfzl9\") pod \"perf-node-gather-daemonset-qfmgf\" (UID: \"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.242687 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.242664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:24.474539 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.474505 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hm75f_81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e/dns/0.log" Sep 18 17:07:24.501458 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.501434 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hm75f_81e2c2c3-c7ae-4ded-980c-c23c9cc4eb1e/kube-rbac-proxy/0.log" Sep 18 17:07:24.513034 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.512897 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf"] Sep 18 17:07:24.515631 ip-10-0-143-79 kubenswrapper[2564]: W0918 17:07:24.515610 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd7dcfc21_05ff_4d7f_823f_8b100b78e0b7.slice/crio-39214a9589eac52d38048be5ce93355634970522685898536ae83072b93f77d5 WatchSource:0}: Error finding container 39214a9589eac52d38048be5ce93355634970522685898536ae83072b93f77d5: Status 404 returned error can't find the container with id 39214a9589eac52d38048be5ce93355634970522685898536ae83072b93f77d5 Sep 18 17:07:24.595273 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:24.595248 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8vqrx_992c20b0-878a-4325-82d7-10fa9f9acd06/dns-node-resolver/0.log" Sep 18 17:07:25.175639 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:25.175611 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wztrg_5e1fc1cd-109e-4bb8-aa37-19c71590b563/node-ca/0.log" Sep 18 17:07:25.384813 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:25.384776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" event={"ID":"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7","Type":"ContainerStarted","Data":"22339fae9255eb30f7e9bddcbead69cfdd30d431ce27ebc77674fc1d9f0d9c1e"} Sep 18 17:07:25.384977 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:25.384860 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" event={"ID":"d7dcfc21-05ff-4d7f-823f-8b100b78e0b7","Type":"ContainerStarted","Data":"39214a9589eac52d38048be5ce93355634970522685898536ae83072b93f77d5"} Sep 18 17:07:25.384977 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:25.384950 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:25.402316 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:25.402270 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" podStartSLOduration=2.402255566 podStartE2EDuration="2.402255566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:07:25.400504065 +0000 UTC m=+3111.919771562" watchObservedRunningTime="2026-09-18 17:07:25.402255566 +0000 UTC m=+3111.921523076" Sep 18 17:07:26.002665 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:26.002624 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-787796f9d4-z4dch_73610849-9bcc-4084-a60e-a0ee6a2a7861/router/0.log" Sep 18 17:07:26.337128 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:26.337055 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-92smk_87108184-50a2-4e70-9887-69a7b01af53f/serve-healthcheck-canary/0.log" Sep 18 17:07:26.755846 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:26.755816 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-6zxcq_b0047f8a-de97-43ae-b9b5-679de1b69eef/insights-operator/0.log" Sep 18 17:07:26.756869 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:26.756851 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-6zxcq_b0047f8a-de97-43ae-b9b5-679de1b69eef/insights-operator/1.log" Sep 18 17:07:26.779630 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:26.779611 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5zlr7_8326ef92-0807-48dd-a612-c5c8113c8591/kube-rbac-proxy/0.log" Sep 18 17:07:26.802439 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:26.802415 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5zlr7_8326ef92-0807-48dd-a612-c5c8113c8591/exporter/0.log" Sep 18 17:07:26.826340 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:26.826316 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5zlr7_8326ef92-0807-48dd-a612-c5c8113c8591/extractor/0.log" Sep 18 17:07:29.220215 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:29.220183 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-dxfp8_9f2bbe5f-3b2b-4490-8f04-eee23d3db2a5/manager/0.log" Sep 18 17:07:31.397546 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:31.397518 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-qfmgf" Sep 18 17:07:33.883837 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:33.883809 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-wbrs4_bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af/kube-storage-version-migrator-operator/1.log" Sep 18 17:07:33.884457 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:33.884443 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-wbrs4_bd3cc907-5ce5-4d5e-89d9-3e60eeb0f1af/kube-storage-version-migrator-operator/0.log" Sep 18 17:07:34.864859 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:34.864833 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cffmd_4ae800b0-36f5-4c25-b07c-cd35de96f0d8/kube-multus-additional-cni-plugins/0.log" Sep 18 17:07:34.889046 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:34.889021 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cffmd_4ae800b0-36f5-4c25-b07c-cd35de96f0d8/egress-router-binary-copy/0.log" Sep 18 17:07:34.911355 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:34.911334 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cffmd_4ae800b0-36f5-4c25-b07c-cd35de96f0d8/cni-plugins/0.log" Sep 18 17:07:34.936922 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:34.936895 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cffmd_4ae800b0-36f5-4c25-b07c-cd35de96f0d8/bond-cni-plugin/0.log" Sep 18 17:07:34.959646 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:34.959623 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cffmd_4ae800b0-36f5-4c25-b07c-cd35de96f0d8/routeoverride-cni/0.log" Sep 18 17:07:34.980616 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:34.980552 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cffmd_4ae800b0-36f5-4c25-b07c-cd35de96f0d8/whereabouts-cni-bincopy/0.log" Sep 18 17:07:35.002810 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:35.002781 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cffmd_4ae800b0-36f5-4c25-b07c-cd35de96f0d8/whereabouts-cni/0.log" Sep 18 17:07:35.349393 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:35.349321 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-f2vz7_931320fc-4a72-46a3-94f2-431ff1c66526/kube-multus/0.log" Sep 18 17:07:35.418449 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:35.418420 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r4pj7_415ad465-e1a1-4c60-99dd-7c18748d64e8/network-metrics-daemon/0.log" Sep 18 17:07:35.444882 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:35.444857 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r4pj7_415ad465-e1a1-4c60-99dd-7c18748d64e8/kube-rbac-proxy/0.log" Sep 18 17:07:36.449513 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:36.449472 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-controller/0.log" Sep 18 17:07:36.469898 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:36.469854 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/1.log" Sep 18 17:07:36.470032 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:36.469931 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovn-acl-logging/0.log" Sep 18 17:07:36.488111 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:36.488088 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/kube-rbac-proxy-node/0.log" Sep 18 17:07:36.510331 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:36.510311 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/kube-rbac-proxy-ovn-metrics/0.log" Sep 18 17:07:36.531365 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:36.531339 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/northd/0.log" Sep 18 17:07:36.553344 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:36.553317 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/nbdb/0.log" Sep 18 17:07:36.575530 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:36.575505 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/sbdb/0.log" Sep 18 17:07:36.613589 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:36.613560 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w9n9q_45cc0900-9bb9-4d4e-9ca6-7f19aa39807f/ovnkube-controller/0.log" Sep 18 17:07:38.004044 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:38.004015 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-z55rv_38551f86-17da-4945-84d1-b8f8cccbcd73/check-endpoints/0.log" Sep 18 17:07:38.035380 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:38.035349 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-7xjxp_02dcbb62-fa0e-491c-9428-59eb7856d6b2/network-check-target-container/0.log" Sep 18 17:07:39.069860 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:39.069826 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-wkmhp_fead61ad-011c-4b2a-81f1-0967e1811606/iptables-alerter/0.log" Sep 18 17:07:39.776597 ip-10-0-143-79 kubenswrapper[2564]: I0918 17:07:39.776572 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9s65v_81ce707b-f655-4f08-aed9-315baa4ac890/tuned/0.log"