Sep 18 16:46:12.833759 ip-10-0-132-99 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 18 16:46:12.833774 ip-10-0-132-99 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 18 16:46:12.833785 ip-10-0-132-99 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 18 16:46:12.834012 ip-10-0-132-99 systemd[1]: Failed to start Kubernetes Kubelet. Sep 18 16:46:23.012396 ip-10-0-132-99 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 18 16:46:23.012414 ip-10-0-132-99 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 734e3e4056594eb598d8080e884d706f -- Sep 18 16:48:32.894732 ip-10-0-132-99 systemd[1]: Starting Kubernetes Kubelet... Sep 18 16:48:36.463508 ip-10-0-132-99 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:48:36.463508 ip-10-0-132-99 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 18 16:48:36.463508 ip-10-0-132-99 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:48:36.463508 ip-10-0-132-99 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 18 16:48:36.463508 ip-10-0-132-99 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:48:36.464173 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.464095 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 18 16:48:36.467974 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.467959 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:48:36.467974 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.467974 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.467979 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.467982 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.467985 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.467988 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.467991 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.467994 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.467997 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468000 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468002 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468005 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468007 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468009 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468012 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468015 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468017 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468020 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468023 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468025 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:48:36.468046 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468028 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468030 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468033 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468035 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468038 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468040 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468042 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468045 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468048 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468050 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468053 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468056 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468060 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468062 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468065 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468068 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468070 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468073 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468075 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468078 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:48:36.468512 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468080 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468082 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468085 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468094 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468096 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468099 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468101 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468103 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468106 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468108 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468111 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468113 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468115 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468118 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468121 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468123 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468126 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468128 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468131 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468133 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:48:36.468995 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468136 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468138 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468140 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468143 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468145 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468148 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468150 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468153 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468155 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468160 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468165 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468169 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468172 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468175 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468177 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468180 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468188 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468190 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468193 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468195 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:48:36.469489 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468197 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468200 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468203 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468205 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468207 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468210 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468213 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468216 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468218 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468221 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468223 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468226 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468230 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468233 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468236 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468238 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468241 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468244 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468247 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:48:36.469979 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468943 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468950 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468953 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468956 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468958 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468961 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468963 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468968 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468971 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468975 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468978 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468980 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468983 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468986 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468989 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468991 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468994 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468997 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.468999 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:48:36.470496 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469003 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469006 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469008 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469011 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469014 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469016 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469019 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469021 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469024 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469027 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469029 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469032 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469034 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469037 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469039 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469042 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469044 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469047 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469049 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:48:36.470978 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469051 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469054 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469056 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469058 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469061 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469063 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469066 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469069 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469071 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469074 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469076 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469078 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469082 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469086 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469089 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469092 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469094 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469097 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469100 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469103 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:48:36.471455 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469105 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469108 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469110 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469113 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469115 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469120 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469123 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469127 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469130 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469133 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469135 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469138 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469141 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469143 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469146 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469149 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469152 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469154 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469157 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:48:36.471941 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469160 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469162 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469164 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469167 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469169 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469171 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469175 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469177 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469180 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469182 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469185 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469187 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469190 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469192 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469195 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469197 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469200 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469202 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469205 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469210 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469213 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:48:36.472389 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469215 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469601 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469607 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469609 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469612 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469615 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469618 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469620 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469624 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469628 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469632 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469635 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469638 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469641 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469644 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469646 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469649 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469652 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469655 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:48:36.472953 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469657 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469660 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469663 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469665 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469668 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469670 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469673 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469675 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469677 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469680 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469682 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469685 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469687 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469690 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469693 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469696 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469698 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469701 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469704 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469706 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:48:36.473438 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469709 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469711 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469714 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469716 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469719 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469722 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469725 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469728 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469730 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469733 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469735 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469737 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469740 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469743 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469745 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469748 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469750 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469753 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469755 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469759 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:48:36.473922 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469763 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469766 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469769 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469772 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469775 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469778 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469781 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469784 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469787 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469790 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469792 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469794 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469797 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469799 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469802 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469805 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469807 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469811 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469814 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469816 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:48:36.474446 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469819 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469821 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469824 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469826 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469829 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469832 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469834 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469836 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469839 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469841 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469844 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469846 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469849 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469851 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469854 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469857 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469859 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469862 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469867 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469870 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:48:36.474940 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.469873 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.469945 2571 flags.go:64] FLAG: --address="0.0.0.0" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.469978 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.469984 2571 flags.go:64] FLAG: --anonymous-auth="true" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.469989 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.469993 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.469997 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470001 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470005 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470008 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470012 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470015 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470019 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470021 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470024 2571 flags.go:64] FLAG: --cgroup-root="" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470027 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470030 2571 flags.go:64] FLAG: --client-ca-file="" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470032 2571 flags.go:64] FLAG: --cloud-provider="external" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470035 2571 flags.go:64] FLAG: --cluster-dns="[]" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470040 2571 flags.go:64] FLAG: --cluster-domain="" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470043 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470045 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470049 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470052 2571 flags.go:64] FLAG: --container-log-max-files="5" Sep 18 16:48:36.475466 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470055 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470058 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470061 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470065 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470068 2571 flags.go:64] FLAG: --contention-profiling="false" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470071 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470078 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470081 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470084 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470088 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470091 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470093 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470096 2571 flags.go:64] FLAG: --enable-load-reader="false" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470099 2571 flags.go:64] FLAG: --enable-server="true" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470102 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470106 2571 flags.go:64] FLAG: --event-burst="100" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470108 2571 flags.go:64] FLAG: --event-qps="50" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470111 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470114 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470118 2571 flags.go:64] FLAG: --eviction-hard="" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470121 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470124 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470126 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470129 2571 flags.go:64] FLAG: --eviction-soft="" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470132 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 18 16:48:36.476046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470134 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470137 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470139 2571 flags.go:64] FLAG: --experimental-mounter-path="" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470142 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470145 2571 flags.go:64] FLAG: --fail-swap-on="true" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470148 2571 flags.go:64] FLAG: --feature-gates="" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470151 2571 flags.go:64] FLAG: --file-check-frequency="20s" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470154 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470157 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470160 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470162 2571 flags.go:64] FLAG: --healthz-port="10248" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470166 2571 flags.go:64] FLAG: --help="false" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470168 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470173 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470176 2571 flags.go:64] FLAG: --http-check-frequency="20s" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470179 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470182 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470185 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470187 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470190 2571 flags.go:64] FLAG: --image-service-endpoint="" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470192 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470195 2571 flags.go:64] FLAG: --kube-api-burst="100" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470198 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470200 2571 flags.go:64] FLAG: --kube-api-qps="50" Sep 18 16:48:36.476675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470203 2571 flags.go:64] FLAG: --kube-reserved="" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470206 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470208 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470212 2571 flags.go:64] FLAG: --kubelet-cgroups="" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470214 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470217 2571 flags.go:64] FLAG: --lock-file="" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470220 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470223 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470225 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470231 2571 flags.go:64] FLAG: --log-json-split-stream="false" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470233 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470236 2571 flags.go:64] FLAG: --log-text-split-stream="false" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470239 2571 flags.go:64] FLAG: --logging-format="text" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470242 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470245 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470248 2571 flags.go:64] FLAG: --manifest-url="" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470250 2571 flags.go:64] FLAG: --manifest-url-header="" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470254 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470257 2571 flags.go:64] FLAG: --max-open-files="1000000" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470261 2571 flags.go:64] FLAG: --max-pods="110" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470264 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470269 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470272 2571 flags.go:64] FLAG: --memory-manager-policy="None" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470275 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470278 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 18 16:48:36.477273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470281 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470284 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470290 2571 flags.go:64] FLAG: --node-status-max-images="50" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470293 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470296 2571 flags.go:64] FLAG: --oom-score-adj="-999" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470299 2571 flags.go:64] FLAG: --pod-cidr="" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470301 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470306 2571 flags.go:64] FLAG: --pod-manifest-path="" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470309 2571 flags.go:64] FLAG: --pod-max-pids="-1" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470312 2571 flags.go:64] FLAG: --pods-per-core="0" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470315 2571 flags.go:64] FLAG: --port="10250" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470318 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470321 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a0a51dad7a63c91c" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470324 2571 flags.go:64] FLAG: --qos-reserved="" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470327 2571 flags.go:64] FLAG: --read-only-port="10255" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470330 2571 flags.go:64] FLAG: --register-node="true" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470333 2571 flags.go:64] FLAG: --register-with-taints="" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470336 2571 flags.go:64] FLAG: --registry-burst="10" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470338 2571 flags.go:64] FLAG: --registry-qps="5" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470341 2571 flags.go:64] FLAG: --reserved-cpus="" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470343 2571 flags.go:64] FLAG: --reserved-memory="" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470347 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470350 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470352 2571 flags.go:64] FLAG: --rotate-certificates="false" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470355 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 18 16:48:36.477933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470358 2571 flags.go:64] FLAG: --runonce="false" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470360 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470363 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470368 2571 flags.go:64] FLAG: --seccomp-default="false" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470370 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470373 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470376 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470379 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470381 2571 flags.go:64] FLAG: --storage-driver-password="root" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470384 2571 flags.go:64] FLAG: --storage-driver-secure="false" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470386 2571 flags.go:64] FLAG: --storage-driver-table="stats" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470389 2571 flags.go:64] FLAG: --storage-driver-user="root" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470392 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470395 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470412 2571 flags.go:64] FLAG: --system-cgroups="" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470415 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470420 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470422 2571 flags.go:64] FLAG: --tls-cert-file="" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470425 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470430 2571 flags.go:64] FLAG: --tls-min-version="" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470433 2571 flags.go:64] FLAG: --tls-private-key-file="" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470435 2571 flags.go:64] FLAG: --topology-manager-policy="none" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470438 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470441 2571 flags.go:64] FLAG: --topology-manager-scope="container" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470444 2571 flags.go:64] FLAG: --v="2" Sep 18 16:48:36.478611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470447 2571 flags.go:64] FLAG: --version="false" Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470451 2571 flags.go:64] FLAG: --vmodule="" Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470454 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.470457 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470578 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470583 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470586 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470589 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470592 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470597 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470600 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470603 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470605 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470608 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470611 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470613 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470616 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470618 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470621 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470624 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:48:36.479606 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470626 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470629 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470632 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470634 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470637 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470639 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470641 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470646 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470650 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470652 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470655 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470657 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470660 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470662 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470665 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470667 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470669 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470672 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470674 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470677 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:48:36.480153 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470679 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470683 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470687 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470689 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470692 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470695 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470698 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470700 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470704 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470707 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470710 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470713 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470716 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470718 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470721 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470724 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470726 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470729 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470731 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:48:36.480658 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470734 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470737 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470739 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470741 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470744 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470746 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470749 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470752 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470754 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470756 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470759 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470761 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470764 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470766 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470770 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470774 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470777 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470779 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470783 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470787 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:48:36.481114 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470789 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470792 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470795 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470797 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470800 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470803 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470805 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470808 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470810 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470813 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470815 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470818 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470822 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470825 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470830 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470833 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470836 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470838 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470841 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470843 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:48:36.481614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470845 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470848 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470851 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.470853 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.471424 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.481108 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.481124 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.481150 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.481156 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481222 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481227 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481230 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481233 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481235 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:48:36.482091 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481238 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481241 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481245 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481247 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481250 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481253 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481256 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481258 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481261 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481264 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481267 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481270 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481274 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481278 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481281 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481284 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481287 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481292 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481295 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:48:36.482474 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481298 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481301 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481304 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481307 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481309 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481312 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481314 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481317 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481320 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481323 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481325 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481328 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481330 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481333 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481335 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481338 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481341 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481343 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481346 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481348 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:48:36.482937 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481351 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481353 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481356 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481358 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481361 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481364 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481366 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481369 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481372 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481374 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481377 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481379 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481382 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481385 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481387 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481390 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481393 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481413 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481418 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:48:36.483437 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481421 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481424 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481427 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481429 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481432 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481434 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481437 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481439 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481442 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481444 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481447 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481449 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481452 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481454 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481458 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481460 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481471 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481475 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481477 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481480 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:48:36.483928 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481482 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481485 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481487 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481490 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481493 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481495 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481497 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481500 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481503 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481505 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481508 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481510 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481513 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481515 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481518 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:48:36.484420 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481520 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.481526 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481655 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481661 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481664 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481666 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481669 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481672 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481675 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481677 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481680 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481683 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481685 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481688 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481697 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:48:36.484783 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481700 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481703 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481705 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481708 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481710 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481713 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481715 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481718 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481720 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481723 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481725 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481728 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481731 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481733 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481735 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481739 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481743 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481746 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481749 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:48:36.485140 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481752 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481754 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481758 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481761 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481764 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481767 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481770 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481773 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481775 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481778 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481780 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481783 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481786 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481789 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481792 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481795 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481797 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481800 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481802 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481805 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:48:36.485637 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481807 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481810 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481812 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481815 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481817 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481820 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481823 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481825 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481828 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481831 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481833 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481836 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481838 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481841 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481843 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481846 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481849 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481851 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481854 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481857 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:48:36.486101 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481860 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481863 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481866 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481868 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481871 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481873 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481882 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481884 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481887 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481890 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481893 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481895 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481898 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481900 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481903 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481905 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481908 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481911 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481914 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481916 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:48:36.486614 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481919 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481921 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481924 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481927 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481929 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481932 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:36.481934 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.481940 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.482047 2571 server.go:962] "Client rotation is on, will bootstrap in background" Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.484187 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.485394 2571 server.go:1019] "Starting client certificate rotation" Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.485508 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:48:36.487113 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.485542 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:48:36.488273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.488259 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 16:48:36.489604 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.489585 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 16:48:36.492863 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.492847 2571 log.go:25] "Validated CRI v1 runtime API" Sep 18 16:48:36.495153 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.495139 2571 log.go:25] "Validated CRI v1 image API" Sep 18 16:48:36.496902 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.496885 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 18 16:48:36.502232 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.502212 2571 fs.go:135] Filesystem UUIDs: map[13d728e8-5990-4963-80c7-cc0f65acb881:/dev/nvme0n1p3 21b885ba-83e5-40d5-b1d8-0dcb82691ba1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 18 16:48:36.502311 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.502232 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 18 16:48:36.503954 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.503929 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:48:36.507549 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.507437 2571 manager.go:216] Machine: {Timestamp:2026-09-18 16:48:36.506884263 +0000 UTC m=+2.029742869 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101383 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d747d07f0c4236adc782c195ee27c SystemUUID:ec2d747d-07f0-c423-6adc-782c195ee27c BootID:734e3e40-5659-4eb5-98d8-080e884d706f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:36:35:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:36:35:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:98:37:c3:71:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 18 16:48:36.507549 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.507545 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 18 16:48:36.507657 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.507647 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 18 16:48:36.509271 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.509253 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 18 16:48:36.509421 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.509273 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 18 16:48:36.510624 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.510614 2571 topology_manager.go:138] "Creating topology manager with none policy" Sep 18 16:48:36.510662 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.510626 2571 container_manager_linux.go:309] "Creating device plugin manager" Sep 18 16:48:36.510662 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.510638 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 16:48:36.510662 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.510648 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 16:48:36.510736 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.510724 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 18 16:48:36.514096 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.514085 2571 state_mem.go:36] "Initialized new in-memory state store" Sep 18 16:48:36.514212 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.514202 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 18 16:48:36.514246 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.514238 2571 kubelet.go:486] "Attempting to sync node with API server" Sep 18 16:48:36.514282 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.514257 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 18 16:48:36.514282 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.514272 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 18 16:48:36.514332 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.514285 2571 kubelet.go:398] "Adding apiserver pod source" Sep 18 16:48:36.514332 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.514297 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 18 16:48:36.514677 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.514663 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 16:48:36.514734 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.514683 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 16:48:36.518569 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.518549 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 18 16:48:36.524263 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.524246 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 18 16:48:36.524346 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.524286 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 18 16:48:36.525559 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525546 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 18 16:48:36.525648 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525567 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 18 16:48:36.525648 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525577 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 18 16:48:36.525648 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525586 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 18 16:48:36.525648 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525599 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 18 16:48:36.525648 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525607 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 18 16:48:36.525648 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525628 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 18 16:48:36.525648 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525637 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 18 16:48:36.525648 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525647 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 18 16:48:36.525927 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525657 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 18 16:48:36.525927 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525670 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 18 16:48:36.525927 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525679 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 18 16:48:36.525927 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525717 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 18 16:48:36.525927 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.525726 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 18 16:48:36.527589 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.527566 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:48:36.527666 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.527567 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:48:36.528442 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.528426 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:36.529701 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.529688 2571 server.go:1268] "Started kubelet" Sep 18 16:48:36.529776 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.529751 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 18 16:48:36.529861 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.529795 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 18 16:48:36.529917 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.529888 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 18 16:48:36.530422 ip-10-0-132-99 systemd[1]: Started Kubernetes Kubelet. Sep 18 16:48:36.531479 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.531463 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 18 16:48:36.531648 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.531635 2571 server.go:310] "Adding debug handlers to kubelet server" Sep 18 16:48:36.533665 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.533641 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 18 16:48:36.537228 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.537215 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 18 16:48:36.537292 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.537232 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 18 16:48:36.537724 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.537527 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:48:36.537905 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.537889 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 18 16:48:36.537998 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.537896 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 18 16:48:36.537998 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.537999 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 18 16:48:36.538158 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.538140 2571 reconstruct.go:97] "Volume reconstruction finished" Sep 18 16:48:36.538158 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.538150 2571 reconciler.go:29] "Reconciler: start to sync state" Sep 18 16:48:36.538273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.538258 2571 factory.go:55] Registering systemd factory Sep 18 16:48:36.538322 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.538280 2571 factory.go:223] Registration of the systemd container factory successfully Sep 18 16:48:36.538546 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.538530 2571 factory.go:153] Registering CRI-O factory Sep 18 16:48:36.538546 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.538545 2571 factory.go:223] Registration of the crio container factory successfully Sep 18 16:48:36.538665 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.538601 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 18 16:48:36.538665 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.538617 2571 factory.go:103] Registering Raw factory Sep 18 16:48:36.538665 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.538627 2571 manager.go:1196] Started watching for new ooms in manager Sep 18 16:48:36.538943 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.538933 2571 manager.go:349] Starting recovery of all containers Sep 18 16:48:36.545964 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.545870 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:48:36.546074 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.545977 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 18 16:48:36.564419 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.564385 2571 manager.go:354] Recovery completed Sep 18 16:48:36.569509 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.569495 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:36.572589 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.572572 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:36.572651 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.572601 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:36.572651 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.572614 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:36.573073 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.573061 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 18 16:48:36.573073 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.573071 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 18 16:48:36.573166 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.573086 2571 state_mem.go:36] "Initialized new in-memory state store" Sep 18 16:48:36.573166 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.573072 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:36.574486 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.574472 2571 policy_none.go:49] "None policy: Start" Sep 18 16:48:36.574544 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.574491 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 18 16:48:36.574544 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.574500 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 18 16:48:36.575799 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.575779 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:36.575866 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.575812 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:36.575866 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.575827 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:36.575997 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.575987 2571 policy_none.go:47] "Start" Sep 18 16:48:36.578322 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.576979 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9c76e01d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.529659933 +0000 UTC m=+2.052518538,LastTimestamp:2026-09-18 16:48:36.529659933 +0000 UTC m=+2.052518538,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.586113 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.586052 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572587518 +0000 UTC m=+2.095446123,LastTimestamp:2026-09-18 16:48:36.572587518 +0000 UTC m=+2.095446123,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.596080 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.596018 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f0637bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572608447 +0000 UTC m=+2.095467052,LastTimestamp:2026-09-18 16:48:36.572608447 +0000 UTC m=+2.095467052,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.604911 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.604843 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f065d6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572618092 +0000 UTC m=+2.095476697,LastTimestamp:2026-09-18 16:48:36.572618092 +0000 UTC m=+2.095476697,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.613194 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.613137 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572587518 +0000 UTC m=+2.095446123,LastTimestamp:2026-09-18 16:48:36.575797781 +0000 UTC m=+2.098656393,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.620331 2571 manager.go:337] "Starting Device Plugin manager" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.620361 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.620370 2571 server.go:87] "Starting device plugin registration server" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.620585 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.620602 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.620691 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.620772 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.620784 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.621238 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.621280 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:48:36.628790 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.622145 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f0637bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f0637bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572608447 +0000 UTC m=+2.095467052,LastTimestamp:2026-09-18 16:48:36.575819608 +0000 UTC m=+2.098678223,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.630414 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.630344 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f065d6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f065d6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572618092 +0000 UTC m=+2.095476697,LastTimestamp:2026-09-18 16:48:36.575832092 +0000 UTC m=+2.098690705,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.639785 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.639728 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783fa20303f1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.622730225 +0000 UTC m=+2.145588816,LastTimestamp:2026-09-18 16:48:36.622730225 +0000 UTC m=+2.145588816,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.685334 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.685304 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 18 16:48:36.686537 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.686523 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 18 16:48:36.686592 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.686573 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 18 16:48:36.686635 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.686595 2571 kubelet.go:2448] "Starting kubelet main sync loop" Sep 18 16:48:36.686635 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.686630 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 18 16:48:36.696756 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.696730 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:48:36.720918 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.720855 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:36.721775 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.721759 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:36.721866 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.721791 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:36.721866 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.721801 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:36.721866 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.721823 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.730806 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.730730 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572587518 +0000 UTC m=+2.095446123,LastTimestamp:2026-09-18 16:48:36.721775106 +0000 UTC m=+2.244633710,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.738489 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.738465 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.738570 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.738469 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f0637bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f0637bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572608447 +0000 UTC m=+2.095467052,LastTimestamp:2026-09-18 16:48:36.721795814 +0000 UTC m=+2.244654420,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.749472 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.749452 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Sep 18 16:48:36.749560 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.749439 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f065d6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f065d6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572618092 +0000 UTC m=+2.095476697,LastTimestamp:2026-09-18 16:48:36.721804633 +0000 UTC m=+2.244663238,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.786864 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.786834 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal"] Sep 18 16:48:36.786940 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.786903 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:36.787976 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.787963 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:36.788062 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.787987 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:36.788062 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.787998 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:36.789328 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.789315 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:36.789473 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.789458 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.789534 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.789494 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:36.790076 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.790056 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:36.790161 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.790091 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:36.790161 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.790102 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:36.790161 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.790058 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:36.790161 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.790158 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:36.790336 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.790168 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:36.790489 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.790432 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572587518 +0000 UTC m=+2.095446123,LastTimestamp:2026-09-18 16:48:36.787974856 +0000 UTC m=+2.310833461,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.791364 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.791349 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.791456 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.791375 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:36.791973 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.791961 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:36.792059 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.791988 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:36.792059 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.792001 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:36.800660 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.800603 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f0637bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f0637bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572608447 +0000 UTC m=+2.095467052,LastTimestamp:2026-09-18 16:48:36.787992236 +0000 UTC m=+2.310850842,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.811828 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.811771 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f065d6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f065d6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572618092 +0000 UTC m=+2.095476697,LastTimestamp:2026-09-18 16:48:36.788001358 +0000 UTC m=+2.310859964,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.820330 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.820267 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572587518 +0000 UTC m=+2.095446123,LastTimestamp:2026-09-18 16:48:36.790077081 +0000 UTC m=+2.312935687,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.826997 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.826979 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.828126 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.828070 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f0637bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f0637bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572608447 +0000 UTC m=+2.095467052,LastTimestamp:2026-09-18 16:48:36.790095602 +0000 UTC m=+2.312954207,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.831394 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.831380 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.837579 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.837524 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f065d6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f065d6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572618092 +0000 UTC m=+2.095476697,LastTimestamp:2026-09-18 16:48:36.790105992 +0000 UTC m=+2.312964597,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.839538 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.839519 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b22dbf502d42c64e57bc1ca69e67914d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.ec2.internal\" (UID: \"b22dbf502d42c64e57bc1ca69e67914d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.839611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.839550 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.839611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.839575 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.846055 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.845997 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572587518 +0000 UTC m=+2.095446123,LastTimestamp:2026-09-18 16:48:36.79014681 +0000 UTC m=+2.313005415,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.853471 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.853390 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f0637bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f0637bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572608447 +0000 UTC m=+2.095467052,LastTimestamp:2026-09-18 16:48:36.790162999 +0000 UTC m=+2.313021602,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.864687 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.864622 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f065d6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f065d6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572618092 +0000 UTC m=+2.095476697,LastTimestamp:2026-09-18 16:48:36.790171915 +0000 UTC m=+2.313030519,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.872732 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.872681 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572587518 +0000 UTC m=+2.095446123,LastTimestamp:2026-09-18 16:48:36.791973636 +0000 UTC m=+2.314832246,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.880360 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.880304 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f0637bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f0637bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572608447 +0000 UTC m=+2.095467052,LastTimestamp:2026-09-18 16:48:36.791994293 +0000 UTC m=+2.314852904,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.891916 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.891856 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f065d6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f065d6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572618092 +0000 UTC m=+2.095476697,LastTimestamp:2026-09-18 16:48:36.792006794 +0000 UTC m=+2.314865400,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.939035 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.939015 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:36.940311 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.940293 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b22dbf502d42c64e57bc1ca69e67914d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.ec2.internal\" (UID: \"b22dbf502d42c64e57bc1ca69e67914d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.940371 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.940320 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.940371 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.940335 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.940459 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.940384 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.940459 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.940391 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b22dbf502d42c64e57bc1ca69e67914d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.ec2.internal\" (UID: \"b22dbf502d42c64e57bc1ca69e67914d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.940459 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.940391 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.942871 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.942853 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:36.942953 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.942885 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:36.942953 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.942895 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:36.942953 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:36.942922 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.952528 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.952465 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f05e5fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572587518 +0000 UTC m=+2.095446123,LastTimestamp:2026-09-18 16:48:36.94287028 +0000 UTC m=+2.465728885,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:36.960071 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.960044 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:36.971389 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:36.971298 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d6783f9f0637bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d6783f9f0637bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:36.572608447 +0000 UTC m=+2.095467052,LastTimestamp:2026-09-18 16:48:36.942890296 +0000 UTC m=+2.465748901,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:37.131220 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.131178 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:48:37.136031 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.136014 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" Sep 18 16:48:37.158295 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:37.158271 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Sep 18 16:48:37.360206 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.360121 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:37.361670 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.361653 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:37.361751 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.361689 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:37.361751 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.361704 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:37.361751 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.361732 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:37.380055 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:37.380032 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:37.474652 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:37.474619 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:48:37.537660 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.537636 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:37.688132 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.688056 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:37.689192 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.689167 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:37.689310 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.689200 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:37.689310 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.689212 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:37.715137 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:37.715111 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:48:37.737854 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:37.737828 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22dbf502d42c64e57bc1ca69e67914d.slice/crio-759d5b14e1e91b6cee658f952c1e8b561710d1689920f53864344d64b06420c2 WatchSource:0}: Error finding container 759d5b14e1e91b6cee658f952c1e8b561710d1689920f53864344d64b06420c2: Status 404 returned error can't find the container with id 759d5b14e1e91b6cee658f952c1e8b561710d1689920f53864344d64b06420c2 Sep 18 16:48:37.738232 ip-10-0-132-99 kubenswrapper[2571]: W0918 16:48:37.738211 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c271c1096af51a1f9f8e05948cf36ce.slice/crio-4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 WatchSource:0}: Error finding container 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875: Status 404 returned error can't find the container with id 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 Sep 18 16:48:37.744494 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:37.744475 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:48:37.753760 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:37.753689 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6783fe4e286cb openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\",Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:37.744674507 +0000 UTC m=+3.267533103,LastTimestamp:2026-09-18 16:48:37.744674507 +0000 UTC m=+3.267533103,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:37.763352 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:37.763293 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-132-99.ec2.internal.18d6783fe4e3a731 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-132-99.ec2.internal,UID:b22dbf502d42c64e57bc1ca69e67914d,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaa467bc105be2f7695ecfea0f0c53f4c5cd52f1f705da378c7a33d13f1d102f\",Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:37.744748337 +0000 UTC m=+3.267606944,LastTimestamp:2026-09-18 16:48:37.744748337 +0000 UTC m=+3.267606944,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:37.778247 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:37.778218 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:48:37.970357 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:37.970279 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Sep 18 16:48:38.033051 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:38.033021 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:48:38.180845 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:38.180814 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:38.181883 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:38.181867 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:38.181955 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:38.181901 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:38.181955 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:38.181917 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:38.181955 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:38.181943 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:38.200282 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:38.200253 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:38.538265 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:38.538230 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:38.691761 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:38.691733 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:48:38.692573 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:38.692525 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" event={"ID":"b22dbf502d42c64e57bc1ca69e67914d","Type":"ContainerStarted","Data":"759d5b14e1e91b6cee658f952c1e8b561710d1689920f53864344d64b06420c2"} Sep 18 16:48:38.693738 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:38.693712 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875"} Sep 18 16:48:38.694193 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:38.694164 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:48:39.456043 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:39.456013 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:48:39.456043 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:39.456013 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:48:39.540605 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:39.540569 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:39.582134 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:39.582090 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Sep 18 16:48:39.678943 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:39.678865 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678405783ea51 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" in 1.923s (1.923s including waiting). Image size: 281584844 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.667853905 +0000 UTC m=+5.190712510,LastTimestamp:2026-09-18 16:48:39.667853905 +0000 UTC m=+5.190712510,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:39.707549 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:39.707477 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-132-99.ec2.internal.18d6784059460ca9 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-132-99.ec2.internal,UID:b22dbf502d42c64e57bc1ca69e67914d,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaa467bc105be2f7695ecfea0f0c53f4c5cd52f1f705da378c7a33d13f1d102f\" in 1.952s (1.952s including waiting). Image size: 320421125 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.697353897 +0000 UTC m=+5.220212489,LastTimestamp:2026-09-18 16:48:39.697353897 +0000 UTC m=+5.220212489,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:39.771611 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:39.771520 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-132-99.ec2.internal.18d678405d2596aa kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-132-99.ec2.internal,UID:b22dbf502d42c64e57bc1ca69e67914d,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Created container: haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.762335402 +0000 UTC m=+5.285193994,LastTimestamp:2026-09-18 16:48:39.762335402 +0000 UTC m=+5.285193994,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:39.780018 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:39.779935 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-132-99.ec2.internal.18d678405d9326ce kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-132-99.ec2.internal,UID:b22dbf502d42c64e57bc1ca69e67914d,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Started container haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.769515726 +0000 UTC m=+5.292374321,LastTimestamp:2026-09-18 16:48:39.769515726 +0000 UTC m=+5.292374321,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:39.801067 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:39.801038 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:39.802101 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:39.802083 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:39.802215 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:39.802121 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:39.802215 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:39.802136 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:39.802215 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:39.802180 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:39.826568 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:39.826539 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:39.925211 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:39.925137 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.914582406 +0000 UTC m=+5.437441019,LastTimestamp:2026-09-18 16:48:39.914582406 +0000 UTC m=+5.437441019,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:39.934031 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:39.933963 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.922070668 +0000 UTC m=+5.444929259,LastTimestamp:2026-09-18 16:48:39.922070668 +0000 UTC m=+5.444929259,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:39.965386 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:39.965336 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:39.966255 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:39.966236 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:39.966318 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:39.966271 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:39.966318 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:39.966281 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:40.537927 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.537899 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:40.545412 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:40.545377 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:48:40.698206 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.698174 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" event={"ID":"b22dbf502d42c64e57bc1ca69e67914d","Type":"ContainerStarted","Data":"4a5018eb28bdd3e8d4978a764394e283e0c67411e973e9d1cffecba9770d19e6"} Sep 18 16:48:40.698206 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.698200 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:40.699050 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.699028 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:40.699165 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.699069 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:40.699165 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.699084 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:40.699286 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:40.699272 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:40.699751 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.699732 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"2ebb87e26937d188c5798e212154b346d6a7088c4dae49db603de2071c24921f"} Sep 18 16:48:40.699789 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.699781 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:40.702137 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.702119 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:40.702292 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.702150 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:40.702292 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:40.702165 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:40.702436 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:40.702312 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:40.894260 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:40.894185 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:48:41.540441 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:41.540416 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:41.701080 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:41.701053 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:41.701494 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:41.701054 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:41.701965 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:41.701946 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:41.702077 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:41.701968 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:41.702077 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:41.701993 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:41.702077 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:41.702006 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:41.702077 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:41.701976 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:41.702077 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:41.702059 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:41.702257 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:41.702230 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:41.702333 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:41.702318 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:42.537615 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:42.537588 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:42.695619 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:42.695576 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:48:42.698234 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:42.698212 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:48:42.791345 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:42.791264 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s" Sep 18 16:48:43.027312 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:43.027264 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:43.028263 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:43.028243 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:43.028339 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:43.028276 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:43.028339 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:43.028285 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:43.028339 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:43.028314 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:43.047374 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:43.047314 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:43.536607 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:43.536527 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:44.540519 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:44.540481 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:44.575837 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:44.575813 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:48:44.772773 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:44.772741 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:44.773926 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:44.773909 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:44.774004 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:44.773943 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:44.774004 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:44.773954 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:44.893204 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:44.893118 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:48:45.032552 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:45.032515 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:48:45.536998 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:45.536972 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:45.961365 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:45.961288 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:48:46.538241 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:46.538212 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:46.621616 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:46.621578 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:48:47.542977 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:47.542946 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:48.540063 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:48.540033 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:49.201115 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:49.201088 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:48:49.448153 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:49.448108 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:49.449127 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:49.449099 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:49.449248 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:49.449142 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:49.449248 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:49.449158 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:49.449248 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:49.449225 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:49.468201 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:49.468140 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:49.537881 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:49.537857 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:50.537559 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:50.537528 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:51.192781 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:51.192620 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:48:51.195260 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:51.195236 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:48:51.538544 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:51.538525 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:51.715784 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:51.715756 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/0.log" Sep 18 16:48:51.715943 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:51.715796 2571 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="2ebb87e26937d188c5798e212154b346d6a7088c4dae49db603de2071c24921f" exitCode=1 Sep 18 16:48:51.715943 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:51.715823 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"2ebb87e26937d188c5798e212154b346d6a7088c4dae49db603de2071c24921f"} Sep 18 16:48:51.716017 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:51.715949 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:51.716833 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:51.716816 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:51.716931 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:51.716847 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:51.716931 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:51.716857 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:51.717034 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:51.717027 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:51.760099 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:51.759228 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784327a87d61 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:51.749846369 +0000 UTC m=+17.272704974,LastTimestamp:2026-09-18 16:48:51.749846369 +0000 UTC m=+17.272704974,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:51.832513 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:51.832438 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.914582406 +0000 UTC m=+5.437441019,LastTimestamp:2026-09-18 16:48:51.822204144 +0000 UTC m=+17.345062752,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:51.842414 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:51.842333 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.922070668 +0000 UTC m=+5.444929259,LastTimestamp:2026-09-18 16:48:51.830560211 +0000 UTC m=+17.353418807,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:48:52.537729 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:52.537690 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:52.719287 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:52.719264 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/0.log" Sep 18 16:48:52.719661 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:52.719304 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"ad87eb60f533ec17b1156c4b9c6187056e57604c5c457b695f377a70478e2c23"} Sep 18 16:48:52.719661 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:52.719426 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:52.720820 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:52.720794 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:52.720914 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:52.720834 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:52.720914 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:52.720850 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:52.721037 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:52.721025 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:53.314535 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:53.314493 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:53.315527 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:53.315511 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:53.315590 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:53.315543 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:53.315590 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:53.315554 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:53.538285 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:53.538249 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:54.272620 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:54.272586 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:48:54.538791 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:54.538714 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:55.538416 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:55.538385 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:56.209651 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:56.209620 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:48:56.469166 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:56.469107 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:48:56.470155 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:56.470133 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:48:56.470237 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:56.470176 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:48:56.470237 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:56.470188 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:48:56.470237 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:56.470215 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:56.487438 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:56.487393 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:48:56.537905 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:56.537880 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:56.622293 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:56.622250 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:48:57.062656 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:57.062619 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:48:57.537998 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:57.537924 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:57.654490 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:57.654459 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:48:58.139614 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:48:58.139584 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:48:58.545704 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:58.545628 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:48:59.537727 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:48:59.537695 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:00.536671 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:00.536638 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:01.539822 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:01.539790 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:02.538589 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:02.538552 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:03.218458 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:03.218397 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:49:03.488515 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.488429 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:03.489419 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.489383 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:03.489496 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.489432 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:03.489496 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.489449 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:03.489496 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.489483 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:03.506775 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:03.506734 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:03.538006 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.537983 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:03.737517 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.737495 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/1.log" Sep 18 16:49:03.737902 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.737885 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/0.log" Sep 18 16:49:03.737960 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.737926 2571 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="ad87eb60f533ec17b1156c4b9c6187056e57604c5c457b695f377a70478e2c23" exitCode=1 Sep 18 16:49:03.737991 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.737958 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"ad87eb60f533ec17b1156c4b9c6187056e57604c5c457b695f377a70478e2c23"} Sep 18 16:49:03.738042 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.738032 2571 scope.go:117] "RemoveContainer" containerID="2ebb87e26937d188c5798e212154b346d6a7088c4dae49db603de2071c24921f" Sep 18 16:49:03.738113 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.738102 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:03.738963 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.738912 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:03.738963 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.738945 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:03.738963 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.738959 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:03.739461 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:03.739174 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:03.739461 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:03.739233 2571 scope.go:117] "RemoveContainer" containerID="2ebb87e26937d188c5798e212154b346d6a7088c4dae49db603de2071c24921f" Sep 18 16:49:03.749250 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:03.749220 2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_0 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: '2ebb87e26937d188c5798e212154b346d6a7088c4dae49db603de2071c24921f'" containerID="2ebb87e26937d188c5798e212154b346d6a7088c4dae49db603de2071c24921f" Sep 18 16:49:03.749332 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:03.749269 2571 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_0 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: '2ebb87e26937d188c5798e212154b346d6a7088c4dae49db603de2071c24921f'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" logger="UnhandledError" Sep 18 16:49:03.749484 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:03.749466 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:49:03.759861 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:03.759779 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:49:04.539438 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:04.539394 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:04.740695 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:04.740671 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/1.log" Sep 18 16:49:05.536673 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:05.536644 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:06.540246 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:06.540217 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:06.622876 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:06.622828 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:49:07.538607 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:07.538582 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:08.536148 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:08.536118 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:08.783467 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:08.783438 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:49:08.786821 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:08.786774 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:49:08.786821 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:08.786801 2571 certificate_manager.go:461] "Reached backoff limit, still unable to rotate certs" err="timed out waiting for the condition" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:49:09.537766 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:09.537738 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:10.229084 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:10.229044 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:49:10.471225 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:10.471184 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:10.472244 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:10.472224 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:10.472391 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:10.472255 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:10.472391 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:10.472265 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:10.507501 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:10.507417 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:10.508668 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:10.508650 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:10.508793 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:10.508686 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:10.508793 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:10.508703 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:10.508793 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:10.508739 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:10.526990 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:10.526965 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:10.536155 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:10.536130 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:10.731764 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:10.731735 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:49:11.539190 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:11.539165 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:12.537073 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:12.537043 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:13.538257 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:13.538226 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:14.538461 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:14.538433 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:14.955049 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:14.955008 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:49:15.153328 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:15.153289 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:49:15.536543 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:15.536516 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:16.539065 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:16.539035 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:16.623489 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:16.623446 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:49:16.687494 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:16.687466 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:16.688308 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:16.688291 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:16.688419 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:16.688320 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:16.688419 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:16.688331 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:16.688531 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:16.688519 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:16.701033 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:16.700944 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784327a87d61\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784327a87d61 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:51.749846369 +0000 UTC m=+17.272704974,LastTimestamp:2026-09-18 16:49:16.689246005 +0000 UTC m=+42.212104610,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:49:16.769728 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:16.769649 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.914582406 +0000 UTC m=+5.437441019,LastTimestamp:2026-09-18 16:49:16.761473163 +0000 UTC m=+42.284331769,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:49:16.779617 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:16.779545 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.922070668 +0000 UTC m=+5.444929259,LastTimestamp:2026-09-18 16:49:16.769620318 +0000 UTC m=+42.292478910,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:49:17.237989 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:17.237957 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:49:17.527526 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.527447 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:17.528473 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.528457 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:17.528553 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.528488 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:17.528553 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.528503 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:17.528553 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.528534 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:17.536729 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.536712 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:17.543387 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:17.543370 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:17.761243 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.761215 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/1.log" Sep 18 16:49:17.761362 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.761258 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"5fe7535fe2b8d95710b3baa7875d6d1fe313e6918ce1f8e33752972632f9e207"} Sep 18 16:49:17.761362 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.761358 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:17.762126 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.762111 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:17.762168 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.762143 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:17.762168 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:17.762155 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:17.762370 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:17.762356 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:18.538782 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:18.538756 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:18.595314 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:18.595290 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:49:19.539747 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:19.539708 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:20.538736 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:20.538708 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:21.545713 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:21.545682 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:22.536920 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:22.536892 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:23.537945 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:23.537911 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:24.248319 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:24.248290 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:49:24.537452 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:24.537384 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:24.543725 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:24.543708 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:24.544584 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:24.544570 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:24.544649 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:24.544598 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:24.544649 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:24.544610 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:24.544649 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:24.544641 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:24.561118 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:24.561096 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:25.538127 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:25.538097 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:26.539105 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:26.539076 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:26.624614 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:26.624567 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:49:27.537415 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:27.537379 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:28.537242 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.537210 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:28.780216 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.780192 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/2.log" Sep 18 16:49:28.780608 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.780593 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/1.log" Sep 18 16:49:28.780663 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.780627 2571 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="5fe7535fe2b8d95710b3baa7875d6d1fe313e6918ce1f8e33752972632f9e207" exitCode=1 Sep 18 16:49:28.780663 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.780651 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"5fe7535fe2b8d95710b3baa7875d6d1fe313e6918ce1f8e33752972632f9e207"} Sep 18 16:49:28.780725 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.780690 2571 scope.go:117] "RemoveContainer" containerID="ad87eb60f533ec17b1156c4b9c6187056e57604c5c457b695f377a70478e2c23" Sep 18 16:49:28.780781 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.780768 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:28.781887 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.781578 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:28.781887 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.781611 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:28.781887 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.781623 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:28.781887 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:28.781818 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:28.781887 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:28.781863 2571 scope.go:117] "RemoveContainer" containerID="ad87eb60f533ec17b1156c4b9c6187056e57604c5c457b695f377a70478e2c23" Sep 18 16:49:28.790747 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:28.790721 2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_1 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: 'ad87eb60f533ec17b1156c4b9c6187056e57604c5c457b695f377a70478e2c23'" containerID="ad87eb60f533ec17b1156c4b9c6187056e57604c5c457b695f377a70478e2c23" Sep 18 16:49:28.790840 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:28.790768 2571 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_1 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: 'ad87eb60f533ec17b1156c4b9c6187056e57604c5c457b695f377a70478e2c23'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" logger="UnhandledError" Sep 18 16:49:28.790944 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:28.790925 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:49:28.805365 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:28.805289 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:49:28.790887634 +0000 UTC m=+54.313746244,Count:2,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:49:29.537681 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:29.537652 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:29.783212 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:29.783190 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/2.log" Sep 18 16:49:30.540336 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:30.540306 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:31.259472 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:31.259435 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:49:31.537734 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:31.537658 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:31.562137 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:31.562116 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:31.563068 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:31.563052 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:31.563109 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:31.563084 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:31.563109 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:31.563094 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:31.563109 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:31.563119 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:31.583150 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:31.583130 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:32.537866 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:32.537841 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:33.539091 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:33.539057 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:34.538212 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:34.538178 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:35.536491 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:35.536462 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:36.540671 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:36.540642 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:36.625389 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:36.625355 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:49:37.536904 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:37.536873 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:38.268902 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:38.268860 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:49:38.539786 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:38.539710 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:38.584245 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:38.584221 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:38.586501 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:38.586479 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:38.586619 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:38.586511 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:38.586619 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:38.586521 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:38.586619 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:38.586547 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:38.602685 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:38.602658 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:39.538917 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:39.538886 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:40.537531 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:40.537506 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:40.687662 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:40.687624 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:40.688885 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:40.688866 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:40.688967 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:40.688900 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:40.688967 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:40.688913 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:40.689109 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:40.689095 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:40.689257 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:40.689242 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:49:40.700788 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:40.700712 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:49:40.689212764 +0000 UTC m=+66.212071360,Count:3,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:49:40.788348 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:40.788281 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:49:40.790411 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:40.790381 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:49:41.538594 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:41.538566 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:42.536850 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:42.536817 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:43.539654 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:43.539625 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:44.426580 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:44.426538 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:44.427675 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:44.427655 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:44.427792 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:44.427689 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:44.427792 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:44.427701 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:44.537089 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:44.537060 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:45.277709 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:45.277676 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:49:45.539843 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:45.539774 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:45.603341 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:45.603306 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:45.604270 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:45.604254 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:45.604382 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:45.604289 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:45.604382 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:45.604304 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:45.604382 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:45.604338 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:45.620489 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:45.620463 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:46.539571 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:46.539481 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:46.626075 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:46.626025 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:49:47.537589 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:47.537563 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:48.538170 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:48.538140 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:49.537992 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:49.537963 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:50.399998 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:50.399961 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:49:50.536834 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:50.536808 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:51.539217 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:51.539187 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:51.687086 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:51.687051 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:49:52.286199 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:52.286164 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:49:52.540459 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:52.540381 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:52.620814 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:52.620772 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:52.621888 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:52.621868 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:52.622008 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:52.621905 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:52.622008 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:52.621920 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:52.622008 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:52.621957 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:52.641757 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:52.641726 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:53.238383 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:53.238347 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:49:53.537927 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:53.537870 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:54.537317 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:54.537285 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:55.541020 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.540992 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:55.686870 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.686831 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:55.687805 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.687782 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:55.687932 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.687816 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:55.687932 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.687833 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:55.688112 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:55.688095 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:55.697716 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:55.697633 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784327a87d61\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784327a87d61 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:51.749846369 +0000 UTC m=+17.272704974,LastTimestamp:2026-09-18 16:49:55.688800174 +0000 UTC m=+81.211658780,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:49:55.766652 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:55.766579 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.914582406 +0000 UTC m=+5.437441019,LastTimestamp:2026-09-18 16:49:55.758241851 +0000 UTC m=+81.281100465,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:49:55.775998 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:55.775926 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.922070668 +0000 UTC m=+5.444929259,LastTimestamp:2026-09-18 16:49:55.766659845 +0000 UTC m=+81.289518436,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:49:55.827824 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.827770 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/2.log" Sep 18 16:49:55.827824 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.827815 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"05563b94ad761b90a8579628b5a8f6720d46e63de712d0392655a1fcbff4fa18"} Sep 18 16:49:55.827935 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.827920 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:55.828613 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.828599 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:55.828676 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.828629 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:55.828676 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:55.828643 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:55.828877 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:55.828866 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:56.536936 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:56.536912 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:56.626806 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:56.626774 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:49:57.538058 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:57.538031 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:58.539591 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:58.539565 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:59.294026 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:59.293997 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:49:59.538509 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:59.538481 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:49:59.642111 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:59.642087 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:49:59.643087 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:59.643063 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:49:59.643209 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:59.643104 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:49:59.643209 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:59.643118 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:49:59.643209 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:49:59.643151 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:49:59.660335 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:49:59.660313 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:00.537234 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:00.537203 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:01.538848 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:01.538818 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:02.538121 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:02.538093 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:03.538933 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:03.538905 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:04.314923 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:04.314883 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:50:04.538009 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:04.537975 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:05.537948 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:05.537916 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:06.305208 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:06.305171 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:50:06.536159 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.536128 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:06.627862 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:06.627834 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:50:06.660803 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.660780 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:06.661623 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.661606 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:06.661697 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.661637 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:06.661697 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.661647 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:06.661697 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.661673 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:06.678496 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:06.678467 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:06.845040 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.845013 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/3.log" Sep 18 16:50:06.845389 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.845375 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/2.log" Sep 18 16:50:06.845465 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.845424 2571 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="05563b94ad761b90a8579628b5a8f6720d46e63de712d0392655a1fcbff4fa18" exitCode=1 Sep 18 16:50:06.845465 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.845453 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"05563b94ad761b90a8579628b5a8f6720d46e63de712d0392655a1fcbff4fa18"} Sep 18 16:50:06.845528 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.845502 2571 scope.go:117] "RemoveContainer" containerID="5fe7535fe2b8d95710b3baa7875d6d1fe313e6918ce1f8e33752972632f9e207" Sep 18 16:50:06.845645 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.845634 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:06.847783 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.847760 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:06.847880 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.847798 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:06.847880 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.847810 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:06.847997 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:06.847985 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:06.848046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:06.848037 2571 scope.go:117] "RemoveContainer" containerID="5fe7535fe2b8d95710b3baa7875d6d1fe313e6918ce1f8e33752972632f9e207" Sep 18 16:50:06.857125 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:06.857099 2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_2 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: '5fe7535fe2b8d95710b3baa7875d6d1fe313e6918ce1f8e33752972632f9e207'" containerID="5fe7535fe2b8d95710b3baa7875d6d1fe313e6918ce1f8e33752972632f9e207" Sep 18 16:50:06.857224 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:06.857146 2571 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_2 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: '5fe7535fe2b8d95710b3baa7875d6d1fe313e6918ce1f8e33752972632f9e207'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" logger="UnhandledError" Sep 18 16:50:06.857327 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:06.857307 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:50:06.869367 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:06.869292 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:50:06.85726596 +0000 UTC m=+92.380124572,Count:4,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:50:07.538594 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:07.538566 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:07.848882 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:07.848812 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/3.log" Sep 18 16:50:08.539006 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:08.538978 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:09.537480 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:09.537450 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:10.536959 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:10.536922 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:10.687171 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:10.687125 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:10.688113 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:10.688089 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:10.688242 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:10.688127 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:10.688242 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:10.688141 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:10.688390 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:10.688376 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:11.537969 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:11.537937 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:12.536765 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:12.536732 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:12.788448 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:12.788364 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:50:12.790782 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:12.790761 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:50:13.314178 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:13.314143 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:50:13.538111 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:13.538080 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:13.679178 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:13.679155 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:13.683122 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:13.683102 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:13.683196 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:13.683143 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:13.683196 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:13.683153 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:13.683196 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:13.683180 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:13.701319 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:13.701291 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:14.538764 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:14.538731 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:15.538033 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:15.538004 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:16.538086 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:16.538052 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:16.628915 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:16.628881 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:50:17.537395 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:17.537367 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:18.537893 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:18.537860 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:19.537855 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:19.537823 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:20.328180 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:20.328145 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:50:20.538292 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:20.538265 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:20.687324 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:20.687297 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:20.688273 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:20.688253 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:20.688368 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:20.688285 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:20.688368 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:20.688295 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:20.688508 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:20.688495 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:20.688647 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:20.688632 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:50:20.696696 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:20.696620 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:50:20.688605233 +0000 UTC m=+106.211463838,Count:5,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:50:20.701714 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:20.701699 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:20.702447 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:20.702433 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:20.702505 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:20.702458 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:20.702505 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:20.702467 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:20.702505 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:20.702489 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:20.719080 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:20.719059 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:21.540343 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:21.540314 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:22.540913 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:22.540886 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:23.538046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:23.538016 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:24.539008 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:24.538983 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:25.538046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:25.538018 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:26.537042 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:26.537014 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:26.629786 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:26.629722 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:50:27.336466 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:27.336436 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:50:27.539279 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:27.539249 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:27.719419 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:27.719363 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:27.720335 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:27.720315 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:27.720424 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:27.720350 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:27.720424 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:27.720360 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:27.720424 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:27.720386 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:27.738459 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:27.738433 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:28.538954 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:28.538926 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:29.536654 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:29.536620 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:30.537470 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:30.537442 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:31.537736 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:31.537709 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:32.537689 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:32.537659 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:32.993900 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:32.993865 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:50:33.538793 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:33.538766 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:34.344858 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:34.344817 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:50:34.537441 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:34.537396 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:34.687418 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:34.687369 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:34.688316 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:34.688300 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:34.688415 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:34.688331 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:34.688415 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:34.688345 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:34.688637 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:34.688621 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:34.688825 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:34.688807 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:50:34.698294 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:34.698212 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:50:34.688771333 +0000 UTC m=+120.211629943,Count:6,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:50:34.738986 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:34.738955 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:34.739937 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:34.739919 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:34.740032 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:34.739950 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:34.740032 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:34.739960 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:34.740032 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:34.739983 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:34.755574 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:34.755549 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:35.538059 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:35.538032 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:36.538468 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:36.538442 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:36.630502 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:36.630470 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:50:37.539922 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:37.539888 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:38.536860 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:38.536832 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:39.536633 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:39.536599 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:40.120195 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:40.120167 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:50:40.538851 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:40.538819 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:41.354977 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:41.354938 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:50:41.538848 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:41.538823 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:41.756596 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:41.756560 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:41.758156 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:41.758137 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:41.758228 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:41.758168 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:41.758228 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:41.758178 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:41.758228 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:41.758205 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:41.773621 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:41.773599 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:42.538186 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:42.538160 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:43.538338 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:43.538308 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:44.539591 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:44.539564 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:44.788436 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:44.788372 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:50:44.790881 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:44.790834 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:50:45.537348 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:45.537315 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:46.538227 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:46.538198 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:46.630812 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:46.630759 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:50:46.693294 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:46.693263 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:50:47.549194 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:47.549163 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:48.363174 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:48.363137 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:50:48.537173 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.537139 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:48.687596 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.687557 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:48.688571 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.688553 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:48.688672 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.688590 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:48.688672 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.688603 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:48.688869 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:48.688854 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:48.699296 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:48.699220 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784327a87d61\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784327a87d61 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:51.749846369 +0000 UTC m=+17.272704974,LastTimestamp:2026-09-18 16:50:48.689660474 +0000 UTC m=+134.212519089,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:50:48.766859 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:48.766767 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.914582406 +0000 UTC m=+5.437441019,LastTimestamp:2026-09-18 16:50:48.758308549 +0000 UTC m=+134.281167164,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:50:48.775231 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.775193 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:48.776326 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.776291 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:48.776468 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.776336 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:48.776468 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.776354 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:48.776468 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.776396 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:48.777446 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:48.777264 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.922070668 +0000 UTC m=+5.444929259,LastTimestamp:2026-09-18 16:50:48.76681584 +0000 UTC m=+134.289674431,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:50:48.796382 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:48.796352 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:48.910853 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.910830 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/3.log" Sep 18 16:50:48.911013 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.910875 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"fa2fa0ae5383dcee184011081a207c4695c90072430a3aeed3b7dca419a6c8c2"} Sep 18 16:50:48.911053 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.911018 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:48.911791 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.911771 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:48.911908 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.911808 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:48.911908 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:48.911823 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:48.912050 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:48.912035 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:49.536595 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:49.536567 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:49.842940 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:49.842866 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:50:50.538805 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:50.538777 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:51.537093 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:51.537066 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:52.538590 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:52.538551 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:53.537980 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:53.537943 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:54.537325 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:54.537292 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:55.014932 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:55.014890 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:55.015977 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:55.015957 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:55.016099 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:55.015986 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:55.016099 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:55.015995 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:55.373090 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:55.373011 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:50:55.536722 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:55.536690 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:55.797037 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:55.797012 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:55.797991 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:55.797968 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:55.798081 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:55.798004 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:55.798081 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:55.798014 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:55.798081 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:55.798040 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:55.816397 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:55.816373 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:56.537128 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:56.537098 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:56.631121 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:56.631078 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:50:57.537784 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:57.537754 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:58.537538 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:58.537507 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:59.537132 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.537100 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:50:59.929695 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.929667 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/4.log" Sep 18 16:50:59.930054 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.930041 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/3.log" Sep 18 16:50:59.930119 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.930072 2571 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="fa2fa0ae5383dcee184011081a207c4695c90072430a3aeed3b7dca419a6c8c2" exitCode=1 Sep 18 16:50:59.930119 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.930098 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"fa2fa0ae5383dcee184011081a207c4695c90072430a3aeed3b7dca419a6c8c2"} Sep 18 16:50:59.930194 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.930139 2571 scope.go:117] "RemoveContainer" containerID="05563b94ad761b90a8579628b5a8f6720d46e63de712d0392655a1fcbff4fa18" Sep 18 16:50:59.930230 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.930210 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:50:59.931018 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.931000 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:50:59.931124 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.931036 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:50:59.931124 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.931050 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:50:59.931536 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:59.931267 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:50:59.931536 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:50:59.931333 2571 scope.go:117] "RemoveContainer" containerID="05563b94ad761b90a8579628b5a8f6720d46e63de712d0392655a1fcbff4fa18" Sep 18 16:50:59.939921 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:59.939898 2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_3 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: '05563b94ad761b90a8579628b5a8f6720d46e63de712d0392655a1fcbff4fa18'" containerID="05563b94ad761b90a8579628b5a8f6720d46e63de712d0392655a1fcbff4fa18" Sep 18 16:50:59.940001 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:59.939941 2571 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_3 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: '05563b94ad761b90a8579628b5a8f6720d46e63de712d0392655a1fcbff4fa18'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" logger="UnhandledError" Sep 18 16:50:59.940117 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:59.940098 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:50:59.949368 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:50:59.949295 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:50:59.940058866 +0000 UTC m=+145.462917478,Count:7,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:51:00.537048 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:00.537016 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:00.932964 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:00.932936 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/4.log" Sep 18 16:51:01.536865 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:01.536834 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:02.384519 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:02.384483 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:51:02.537700 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:02.537665 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:02.817313 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:02.817225 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:02.818242 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:02.818215 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:02.818368 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:02.818255 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:02.818368 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:02.818267 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:02.818368 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:02.818297 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:02.836191 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:02.836161 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:03.538260 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:03.538228 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:04.536684 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:04.536654 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:05.538202 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:05.538173 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:06.539831 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:06.539802 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:06.631657 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:06.631608 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:51:07.537011 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:07.536982 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:08.543203 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:08.543164 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:09.393764 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:09.393726 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:51:09.537392 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:09.537361 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:09.836756 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:09.836688 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:09.837740 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:09.837719 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:09.837827 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:09.837755 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:09.837827 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:09.837766 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:09.837827 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:09.837791 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:09.856062 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:09.856038 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:10.537729 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:10.537696 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:10.577698 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:10.577667 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:51:11.537935 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:11.537904 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:12.540069 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:12.540039 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:13.539800 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:13.539771 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:14.537150 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:14.537118 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:14.687430 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:14.687372 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:14.688418 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:14.688379 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:14.688503 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:14.688431 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:14.688503 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:14.688444 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:14.688641 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:14.688629 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:14.688804 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:14.688787 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:51:14.697233 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:14.697153 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:51:14.688739362 +0000 UTC m=+160.211597967,Count:8,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:51:15.538975 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:15.538938 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:16.402126 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:16.402059 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:51:16.536493 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:16.536467 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:16.632294 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:16.632250 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:51:16.789028 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:16.788936 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:51:16.791277 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:16.791254 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:51:16.856643 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:16.856625 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:16.857578 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:16.857558 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:16.857649 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:16.857590 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:16.857649 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:16.857600 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:16.857649 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:16.857626 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:16.880122 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:16.880102 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:17.536860 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:17.536829 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:18.539925 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:18.539898 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:19.537489 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:19.537462 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:20.538612 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:20.538585 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:21.538571 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:21.538542 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:22.537462 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:22.537437 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:23.411759 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:23.411729 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:51:23.538291 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:23.538261 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:23.881126 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:23.881100 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:23.882090 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:23.882075 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:23.882169 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:23.882104 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:23.882169 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:23.882114 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:23.882169 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:23.882138 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:23.901562 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:23.901537 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:24.538785 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:24.538757 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:25.538382 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:25.538348 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:25.806200 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:25.806137 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:51:26.539142 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:26.539111 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:26.632814 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:26.632773 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:51:27.538195 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:27.538168 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:27.687415 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:27.687371 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:27.688323 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:27.688305 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:27.688396 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:27.688339 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:27.688396 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:27.688354 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:27.688596 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:27.688582 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:27.688742 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:27.688727 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:51:27.697162 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:27.697092 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:51:27.688697959 +0000 UTC m=+173.211556563,Count:9,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:51:28.538901 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:28.538874 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:29.537265 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:29.537235 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:30.420516 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:30.420487 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:51:30.539983 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:30.539955 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:30.902559 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:30.902529 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:30.903546 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:30.903525 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:30.903621 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:30.903560 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:30.903621 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:30.903572 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:30.903621 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:30.903606 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:30.921129 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:30.921091 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:31.539214 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:31.539188 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:32.538348 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:32.538321 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:33.143434 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:33.143381 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:51:33.538491 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:33.538396 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:33.687326 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:33.687278 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:33.688413 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:33.688384 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:33.688479 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:33.688432 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:33.688479 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:33.688443 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:33.688650 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:33.688637 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:34.538858 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:34.538827 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:35.537560 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:35.537530 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:36.539864 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:36.539835 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:36.633413 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:36.633368 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:51:37.073483 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:37.073453 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:51:37.430145 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:37.430117 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:51:37.538352 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:37.538325 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:37.921320 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:37.921298 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:37.922344 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:37.922316 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:37.922344 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:37.922347 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:37.922509 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:37.922359 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:37.922509 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:37.922384 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:37.939293 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:37.939269 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:38.536959 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:38.536928 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:39.539483 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:39.539453 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:40.537046 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:40.537013 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:41.536739 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:41.536707 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:42.538750 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:42.538716 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:42.686831 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:42.686785 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:42.688655 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:42.688635 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:42.688765 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:42.688671 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:42.688765 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:42.688685 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:42.688960 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:42.688944 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:42.689127 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:42.689107 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:51:42.700806 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:42.700720 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:51:42.689076902 +0000 UTC m=+188.211935494,Count:10,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:51:43.537289 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:43.537255 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:44.443192 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:44.443152 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:51:44.539277 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:44.539253 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:44.939503 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:44.939469 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:44.941528 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:44.941503 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:44.941623 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:44.941542 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:44.941623 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:44.941555 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:44.941623 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:44.941583 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:44.960584 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:44.960562 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:45.537593 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:45.537572 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:46.537018 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:46.536987 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:46.633971 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:46.633942 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:51:47.541225 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:47.541194 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:48.539596 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:48.539572 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:48.788030 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:48.787990 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:51:48.790642 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:48.790595 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:51:49.539524 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:49.539488 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:49.847295 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:49.847217 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:51:50.537259 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:50.537227 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:51.450728 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:51.450694 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:51:51.538243 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:51.538210 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:51.961069 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:51.961038 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:51.962396 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:51.962379 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:51.962533 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:51.962436 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:51.962533 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:51.962448 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:51.962533 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:51.962475 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:51.979664 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:51.979637 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:52.538141 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:52.538106 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:53.539772 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:53.539740 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:54.537065 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:54.537033 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:55.538209 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:55.538178 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:55.687229 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:55.687187 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:55.688164 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:55.688147 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:55.688261 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:55.688181 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:55.688261 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:55.688195 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:55.688489 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:55.688474 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:55.688659 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:55.688640 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:51:55.697191 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:55.697109 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:51:55.688607784 +0000 UTC m=+201.211466388,Count:11,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:51:56.538493 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:56.538463 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:56.635068 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:56.635033 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:51:57.538452 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:57.538395 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:58.463600 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:58.463567 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:51:58.536939 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:58.536912 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:51:58.980413 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:58.980381 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:51:58.981372 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:58.981356 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:51:58.981454 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:58.981390 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:51:58.981454 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:58.981420 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:51:58.981454 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:58.981451 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:58.999157 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:51:58.999137 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:51:59.540414 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:51:59.540370 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:00.538222 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:00.538196 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:01.343270 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:01.343235 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:52:01.540218 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:01.540189 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:02.538489 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:02.538458 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:03.537629 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:03.537597 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:04.539854 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:04.539825 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:05.474388 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:05.474351 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:52:05.537652 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:05.537626 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:06.000076 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:06.000040 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:06.001068 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:06.001050 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:06.001162 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:06.001081 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:06.001162 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:06.001092 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:06.001162 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:06.001119 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:06.019462 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:06.019441 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:06.537973 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:06.537945 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:06.635903 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:06.635869 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:52:06.686866 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:06.686844 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:06.687911 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:06.687895 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:06.688002 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:06.687924 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:06.688002 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:06.687939 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:06.688152 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:06.688140 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:06.688291 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:06.688274 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:52:06.697244 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:06.697177 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:52:06.688247921 +0000 UTC m=+212.211106525,Count:12,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:52:07.537887 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:07.537853 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:08.535949 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:08.535920 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:09.539129 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:09.539097 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:10.538550 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:10.538515 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:11.538082 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:11.538047 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:12.489328 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:12.489294 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:52:12.540048 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:12.540023 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:13.020467 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:13.020429 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:13.021450 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:13.021428 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:13.021562 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:13.021464 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:13.021562 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:13.021474 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:13.021562 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:13.021499 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:13.037126 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:13.037098 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:13.538742 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:13.538710 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:14.538869 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:14.538835 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:15.538355 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:15.538324 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:16.538928 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:16.538899 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:16.636374 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:16.636332 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:52:17.536697 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:17.536669 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:18.060277 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:18.060244 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:52:18.539010 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:18.538978 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:19.499363 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:19.499325 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:52:19.536496 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:19.536471 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:19.687123 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:19.687097 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:19.688056 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:19.688037 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:19.688170 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:19.688069 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:19.688170 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:19.688081 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:19.688298 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:19.688286 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:19.688462 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:19.688446 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:52:19.697332 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:19.697262 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:52:19.688416898 +0000 UTC m=+225.211275503,Count:13,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:52:20.037840 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:20.037812 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:20.039424 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:20.039392 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:20.039520 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:20.039436 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:20.039520 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:20.039449 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:20.039520 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:20.039476 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:20.057368 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:20.057337 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:20.538364 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:20.538336 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:20.788875 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:20.788800 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:52:20.791663 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:20.791641 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:52:21.536937 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:21.536909 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:22.536257 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:22.536224 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:23.538600 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:23.538573 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:24.537101 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:24.537069 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:25.537138 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:25.537108 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:26.509479 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:26.509446 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:52:26.536646 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:26.536625 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:26.637140 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:26.637110 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:52:27.057713 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:27.057686 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:27.058510 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:27.058493 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:27.058611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:27.058524 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:27.058611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:27.058533 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:27.058611 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:27.058560 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:27.079770 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:27.079746 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:27.536270 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:27.536243 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:27.616379 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:27.616349 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:52:28.538276 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:28.538245 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:29.536324 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:29.536294 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:30.538905 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:30.538870 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:30.687785 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:30.687740 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:30.688738 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:30.688719 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:30.688839 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:30.688753 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:30.688839 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:30.688764 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:30.688985 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:30.688972 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:30.699297 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:30.699195 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784327a87d61\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784327a87d61 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:51.749846369 +0000 UTC m=+17.272704974,LastTimestamp:2026-09-18 16:52:30.689707428 +0000 UTC m=+236.212566021,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:52:30.766540 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:30.766472 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d678406638b186 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.914582406 +0000 UTC m=+5.437441019,LastTimestamp:2026-09-18 16:52:30.757703011 +0000 UTC m=+236.280561620,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:52:30.774102 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:30.774037 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6784066aaf48c openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:48:39.922070668 +0000 UTC m=+5.444929259,LastTimestamp:2026-09-18 16:52:30.765854349 +0000 UTC m=+236.288712943,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:52:31.062735 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:31.062704 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/4.log" Sep 18 16:52:31.062910 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:31.062756 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"b915e7a912e31663470c39906dfe99e76383d8c39daada22ec99a66075623730"} Sep 18 16:52:31.062910 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:31.062889 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:31.063718 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:31.063691 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:31.063818 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:31.063727 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:31.063818 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:31.063742 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:31.063913 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:31.063901 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:31.538367 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:31.538338 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:32.536622 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:32.536596 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:33.520167 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:33.520134 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:52:33.538976 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:33.538954 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:34.080206 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:34.080183 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:34.081534 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:34.081517 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:34.081628 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:34.081547 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:34.081628 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:34.081557 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:34.081628 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:34.081582 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:34.098932 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:34.098907 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:34.267568 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:34.267537 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:52:34.537279 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:34.537255 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:35.536431 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:35.536387 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:36.542004 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:36.541977 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:36.637911 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:36.637878 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:52:37.537024 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:37.536992 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:38.538967 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:38.538941 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:39.537442 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:39.537388 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:40.527855 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:40.527819 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:52:40.537670 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:40.537651 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:41.099438 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:41.099396 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:41.100444 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:41.100424 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:41.100545 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:41.100459 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:41.100545 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:41.100470 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:41.100545 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:41.100497 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:41.115790 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:41.115770 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:41.538692 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:41.538668 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:42.079290 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.079262 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/5.log" Sep 18 16:52:42.079660 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.079643 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/4.log" Sep 18 16:52:42.079733 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.079675 2571 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="b915e7a912e31663470c39906dfe99e76383d8c39daada22ec99a66075623730" exitCode=1 Sep 18 16:52:42.079733 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.079698 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"b915e7a912e31663470c39906dfe99e76383d8c39daada22ec99a66075623730"} Sep 18 16:52:42.079803 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.079764 2571 scope.go:117] "RemoveContainer" containerID="fa2fa0ae5383dcee184011081a207c4695c90072430a3aeed3b7dca419a6c8c2" Sep 18 16:52:42.079849 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.079838 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:42.082468 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.082444 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:42.082539 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.082475 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:42.082539 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.082488 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:42.082658 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:42.082647 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:42.082705 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.082692 2571 scope.go:117] "RemoveContainer" containerID="fa2fa0ae5383dcee184011081a207c4695c90072430a3aeed3b7dca419a6c8c2" Sep 18 16:52:42.091760 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:42.091733 2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_4 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: 'fa2fa0ae5383dcee184011081a207c4695c90072430a3aeed3b7dca419a6c8c2'" containerID="fa2fa0ae5383dcee184011081a207c4695c90072430a3aeed3b7dca419a6c8c2" Sep 18 16:52:42.091871 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:42.091780 2571 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_4 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: 'fa2fa0ae5383dcee184011081a207c4695c90072430a3aeed3b7dca419a6c8c2'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" logger="UnhandledError" Sep 18 16:52:42.091964 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:42.091945 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:52:42.100765 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:42.100692 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:52:42.091900296 +0000 UTC m=+247.614758920,Count:14,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:52:42.538535 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:42.538509 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:43.082891 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:43.082865 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/5.log" Sep 18 16:52:43.540534 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:43.540508 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:44.538582 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:44.538550 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:45.537981 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:45.537950 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:46.540209 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:46.540135 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:46.638491 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:46.638452 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:52:47.536575 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:47.536544 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:47.536575 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:47.536549 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:52:48.116660 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:48.116632 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:48.118983 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:48.118965 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:48.119083 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:48.118996 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:48.119083 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:48.119005 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:48.119083 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:48.119030 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:48.138752 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:48.138724 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:48.537820 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:48.537797 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:48.687363 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:48.687326 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:48.688578 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:48.688557 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:48.688687 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:48.688594 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:48.688687 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:48.688610 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:48.688859 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:48.688844 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:49.538069 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:49.538039 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:50.538121 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:50.538091 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:51.536608 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:51.536576 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:52.537234 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:52.537199 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:52.788234 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:52.788156 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:52:52.790410 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:52.790375 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:52:53.539327 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:53.539298 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:54.537724 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:54.537688 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:54.544720 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:54.544694 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:52:55.139777 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:55.139741 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:55.141252 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:55.141235 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:55.141344 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:55.141266 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:55.141344 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:55.141275 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:55.141344 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:55.141301 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:55.161158 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:55.161136 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:55.537772 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:55.537705 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:56.538561 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:56.538527 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:56.638976 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:56.638939 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:52:56.686886 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:56.686859 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:52:56.687679 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:56.687659 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:52:56.687792 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:56.687694 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:52:56.687792 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:56.687709 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:52:56.687942 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:56.687927 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:52:56.688117 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:56.688101 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:52:56.696876 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:56.696803 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:52:56.688066924 +0000 UTC m=+262.210925529,Count:15,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:52:57.536825 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:57.536790 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:58.110150 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:52:58.110118 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:52:58.536705 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:58.536679 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:52:59.536269 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:52:59.536236 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:00.539798 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:00.539767 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:01.538032 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:01.538000 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:01.554072 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:01.554045 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:53:02.161472 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:02.161437 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:02.162490 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:02.162473 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:02.162590 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:02.162503 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:02.162590 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:02.162513 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:02.162590 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:02.162540 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:02.179584 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:02.179558 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:02.537149 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:02.537078 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:03.538309 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:03.538277 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:04.538702 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:04.538665 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:05.512977 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:05.512938 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:53:05.538242 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:05.538215 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:06.539576 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:06.539546 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:06.639078 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:06.639037 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:53:07.536953 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:07.536923 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:08.160211 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:08.160183 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:08.161069 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:08.161051 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:08.161147 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:08.161082 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:08.161147 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:08.161095 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:08.537228 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:08.537157 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:08.563601 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:08.563570 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:53:08.687123 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:08.687093 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:08.691749 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:08.691732 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:08.691844 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:08.691762 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:08.691844 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:08.691774 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:08.691986 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:08.691975 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:08.692124 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:08.692110 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:53:08.701500 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:08.701431 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:53:08.692081248 +0000 UTC m=+274.214939853,Count:16,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:09.179700 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:09.179665 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:09.180607 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:09.180589 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:09.180683 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:09.180622 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:09.180683 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:09.180631 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:09.180683 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:09.180657 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:09.197009 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:09.196988 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:09.538542 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:09.538477 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:10.537650 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:10.537618 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:10.873690 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:10.873619 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:53:11.540711 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:11.540684 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:12.538576 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:12.538545 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:13.536815 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:13.536789 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:14.538030 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:14.537998 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:15.536854 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:15.536823 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:15.574004 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:15.573970 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:53:15.895684 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:15.895650 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:53:16.197160 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:16.197076 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:16.198015 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:16.197997 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:16.198101 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:16.198028 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:16.198101 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:16.198037 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:16.198101 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:16.198061 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:16.213622 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:16.213599 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:16.537968 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:16.537903 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:16.639783 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:16.639755 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:53:17.537868 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:17.537835 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:18.537613 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:18.537572 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:19.538921 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:19.538893 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:20.541937 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:20.541897 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:20.687313 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:20.687273 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:20.688265 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:20.688247 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:20.688329 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:20.688308 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:20.688329 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:20.688320 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:20.688551 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:20.688538 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:20.688694 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:20.688679 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:53:20.698878 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:20.698800 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:53:20.688652486 +0000 UTC m=+286.211511078,Count:17,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:21.540906 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:21.539220 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:22.538113 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:22.538076 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:22.583879 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:22.583849 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:53:23.214481 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:23.214434 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:23.215455 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:23.215439 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:23.215522 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:23.215472 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:23.215522 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:23.215488 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:23.215522 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:23.215520 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:23.233353 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:23.233329 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:23.537156 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:23.537090 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:24.538672 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:24.538636 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:24.788638 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:24.788604 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:53:24.791130 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:24.791061 2571 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:53:25.537440 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:25.537393 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:26.538347 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:26.538318 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:26.639919 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:26.639884 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:53:27.539084 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:27.539047 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:28.538867 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:28.538837 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:29.541206 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:29.541167 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:29.594840 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:29.594806 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:53:30.233585 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:30.233555 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:30.234496 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:30.234478 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:30.234612 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:30.234514 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:30.234612 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:30.234528 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:30.234612 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:30.234563 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:30.252545 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:30.252515 2571 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:30.538329 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:30.538259 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:31.537451 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:31.537417 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:32.537454 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:32.537420 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:33.538787 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:33.538750 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:34.537628 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:34.537599 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:34.687615 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:34.687586 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:34.688559 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:34.688543 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:34.688636 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:34.688572 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:34.688636 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:34.688584 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:34.688794 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:34.688783 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:34.688928 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:34.688913 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:53:34.696844 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:34.696770 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67845f2e3a1ac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:49:03.749431724 +0000 UTC m=+29.272290329,LastTimestamp:2026-09-18 16:53:34.688886983 +0000 UTC m=+300.211745574,Count:18,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:35.538315 ip-10-0-132-99 kubenswrapper[2571]: I0918 16:53:35.538287 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:36.484717 ip-10-0-132-99 kubenswrapper[2571]: E0918 16:53:36.484642 2571 transport.go:120] "No valid client certificate is found and the server is responsive, exiting." lastCertificateAvailabilityTime="2026-09-18 16:48:36.484304703 +0000 UTC m=+2.007163314" shutdownThreshold="5m0s" Sep 18 16:53:36.506654 ip-10-0-132-99 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE Sep 18 16:53:36.506780 ip-10-0-132-99 systemd[1]: kubelet.service: Failed with result 'exit-code'. Sep 18 16:53:36.507018 ip-10-0-132-99 systemd[1]: kubelet.service: Consumed 3.198s CPU time. Sep 18 16:53:46.627477 ip-10-0-132-99 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1. Sep 18 16:53:46.627639 ip-10-0-132-99 systemd[1]: Stopped Kubernetes Kubelet. Sep 18 16:53:46.627693 ip-10-0-132-99 systemd[1]: kubelet.service: Consumed 3.198s CPU time. Sep 18 16:53:46.636727 ip-10-0-132-99 systemd[1]: Starting Kubernetes Kubelet... Sep 18 16:53:46.694800 ip-10-0-132-99 kubenswrapper[2874]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:53:46.694800 ip-10-0-132-99 kubenswrapper[2874]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 18 16:53:46.694800 ip-10-0-132-99 kubenswrapper[2874]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:53:46.694800 ip-10-0-132-99 kubenswrapper[2874]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 18 16:53:46.694800 ip-10-0-132-99 kubenswrapper[2874]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:53:46.695249 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.694863 2874 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 18 16:53:46.696679 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696664 2874 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:53:46.696679 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696680 2874 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696683 2874 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696686 2874 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696689 2874 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696692 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696694 2874 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696697 2874 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696699 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696702 2874 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696705 2874 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696707 2874 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696710 2874 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696712 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696715 2874 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696717 2874 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696720 2874 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696722 2874 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696725 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696728 2874 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696737 2874 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:53:46.696749 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696740 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696743 2874 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696746 2874 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696748 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696751 2874 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696753 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696756 2874 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696759 2874 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696762 2874 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696764 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696769 2874 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696773 2874 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696777 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696780 2874 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696783 2874 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696786 2874 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696788 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696792 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696794 2874 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:53:46.697258 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696797 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696799 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696802 2874 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696805 2874 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696807 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696809 2874 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696812 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696814 2874 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696817 2874 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696819 2874 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696822 2874 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696824 2874 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696827 2874 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696829 2874 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696833 2874 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696837 2874 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696840 2874 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696843 2874 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696846 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:53:46.697760 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696849 2874 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696851 2874 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696854 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696856 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696859 2874 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696862 2874 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696864 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696867 2874 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696870 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696872 2874 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696875 2874 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696877 2874 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696880 2874 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696882 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696886 2874 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696888 2874 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696891 2874 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696894 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696896 2874 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696898 2874 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:53:46.698239 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696901 2874 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696903 2874 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696905 2874 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696908 2874 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696910 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696913 2874 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696915 2874 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696918 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696922 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696924 2874 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696927 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696929 2874 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696932 2874 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696934 2874 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696937 2874 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696939 2874 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696942 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696944 2874 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696946 2874 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.696949 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:53:46.698807 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697628 2874 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697635 2874 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697638 2874 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697641 2874 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697643 2874 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697654 2874 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697658 2874 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697661 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697664 2874 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697666 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697670 2874 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697672 2874 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697675 2874 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697678 2874 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697680 2874 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697683 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697685 2874 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697688 2874 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697690 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697693 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:53:46.699351 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697696 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697699 2874 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697702 2874 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697705 2874 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697707 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697710 2874 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697712 2874 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697715 2874 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697717 2874 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697719 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697722 2874 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697724 2874 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697727 2874 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697729 2874 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697733 2874 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697736 2874 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697738 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697741 2874 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697744 2874 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:53:46.699889 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697747 2874 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697750 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697752 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697754 2874 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697757 2874 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697759 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697762 2874 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697764 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697767 2874 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697770 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697773 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697775 2874 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697778 2874 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697781 2874 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697784 2874 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697786 2874 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697789 2874 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697791 2874 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697794 2874 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697796 2874 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:53:46.700386 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697799 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697801 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697804 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697806 2874 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697808 2874 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697811 2874 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697814 2874 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697816 2874 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697818 2874 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697821 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697823 2874 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697826 2874 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697829 2874 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697831 2874 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697834 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697836 2874 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697838 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697841 2874 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697843 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:53:46.700901 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697846 2874 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697849 2874 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697851 2874 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697855 2874 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697858 2874 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697861 2874 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697865 2874 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697868 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697871 2874 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697873 2874 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697876 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697878 2874 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697881 2874 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697884 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697886 2874 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697889 2874 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697891 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697894 2874 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697896 2874 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697898 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:53:46.701381 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.697901 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698235 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698241 2874 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698244 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698247 2874 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698250 2874 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698253 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698255 2874 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698258 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698261 2874 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698263 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698265 2874 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698268 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698271 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698273 2874 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698276 2874 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698278 2874 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698281 2874 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698286 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698288 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:53:46.701953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698291 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698295 2874 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698299 2874 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698302 2874 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698305 2874 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698308 2874 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698311 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698313 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698316 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698318 2874 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698321 2874 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698323 2874 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698326 2874 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698328 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698330 2874 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698333 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698335 2874 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698338 2874 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698340 2874 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:53:46.702475 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698343 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698345 2874 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698348 2874 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698350 2874 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698353 2874 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698355 2874 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698357 2874 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698360 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698362 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698366 2874 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698368 2874 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698372 2874 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698375 2874 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698377 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698380 2874 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698382 2874 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698385 2874 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698387 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698390 2874 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698392 2874 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:53:46.703284 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698395 2874 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698412 2874 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698415 2874 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698418 2874 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698427 2874 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698431 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698434 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698437 2874 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698440 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698442 2874 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698445 2874 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698447 2874 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698450 2874 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698453 2874 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698456 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698458 2874 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698461 2874 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698463 2874 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698465 2874 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698468 2874 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:53:46.704182 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698471 2874 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698474 2874 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698476 2874 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698481 2874 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698483 2874 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698486 2874 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698488 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698491 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698493 2874 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698496 2874 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698500 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698503 2874 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698505 2874 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698508 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698510 2874 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698512 2874 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698515 2874 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698517 2874 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698520 2874 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698522 2874 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:53:46.704976 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.698525 2874 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698592 2874 flags.go:64] FLAG: --address="0.0.0.0" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698599 2874 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698605 2874 flags.go:64] FLAG: --anonymous-auth="true" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698609 2874 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698614 2874 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698617 2874 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698621 2874 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698625 2874 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698628 2874 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698631 2874 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698634 2874 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698637 2874 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698642 2874 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698645 2874 flags.go:64] FLAG: --cgroup-root="" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698649 2874 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698652 2874 flags.go:64] FLAG: --client-ca-file="" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698655 2874 flags.go:64] FLAG: --cloud-provider="external" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698658 2874 flags.go:64] FLAG: --cluster-dns="[]" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698662 2874 flags.go:64] FLAG: --cluster-domain="" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698664 2874 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698667 2874 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698670 2874 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698674 2874 flags.go:64] FLAG: --container-log-max-files="5" Sep 18 16:53:46.705543 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698677 2874 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698680 2874 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698683 2874 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698686 2874 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698689 2874 flags.go:64] FLAG: --contention-profiling="false" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698692 2874 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698694 2874 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698697 2874 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698700 2874 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698703 2874 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698706 2874 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698709 2874 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698712 2874 flags.go:64] FLAG: --enable-load-reader="false" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698715 2874 flags.go:64] FLAG: --enable-server="true" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698718 2874 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698722 2874 flags.go:64] FLAG: --event-burst="100" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698726 2874 flags.go:64] FLAG: --event-qps="50" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698728 2874 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698731 2874 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698734 2874 flags.go:64] FLAG: --eviction-hard="" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698737 2874 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698741 2874 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698744 2874 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698748 2874 flags.go:64] FLAG: --eviction-soft="" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698750 2874 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 18 16:53:46.706148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698753 2874 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698756 2874 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698759 2874 flags.go:64] FLAG: --experimental-mounter-path="" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698762 2874 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698765 2874 flags.go:64] FLAG: --fail-swap-on="true" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698768 2874 flags.go:64] FLAG: --feature-gates="" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698771 2874 flags.go:64] FLAG: --file-check-frequency="20s" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698774 2874 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698777 2874 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698780 2874 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698783 2874 flags.go:64] FLAG: --healthz-port="10248" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698786 2874 flags.go:64] FLAG: --help="false" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698788 2874 flags.go:64] FLAG: --hostname-override="ip-10-0-132-99.ec2.internal" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698791 2874 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698794 2874 flags.go:64] FLAG: --http-check-frequency="20s" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698797 2874 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698800 2874 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698803 2874 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698806 2874 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698809 2874 flags.go:64] FLAG: --image-service-endpoint="" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698812 2874 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698814 2874 flags.go:64] FLAG: --kube-api-burst="100" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698819 2874 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698822 2874 flags.go:64] FLAG: --kube-api-qps="50" Sep 18 16:53:46.706796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698825 2874 flags.go:64] FLAG: --kube-reserved="" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698827 2874 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698830 2874 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698833 2874 flags.go:64] FLAG: --kubelet-cgroups="" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698837 2874 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698840 2874 flags.go:64] FLAG: --lock-file="" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698843 2874 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698846 2874 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698849 2874 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698854 2874 flags.go:64] FLAG: --log-json-split-stream="false" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698857 2874 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698859 2874 flags.go:64] FLAG: --log-text-split-stream="false" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698862 2874 flags.go:64] FLAG: --logging-format="text" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698865 2874 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698868 2874 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698871 2874 flags.go:64] FLAG: --manifest-url="" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698874 2874 flags.go:64] FLAG: --manifest-url-header="" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698878 2874 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698881 2874 flags.go:64] FLAG: --max-open-files="1000000" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698884 2874 flags.go:64] FLAG: --max-pods="110" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698887 2874 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698890 2874 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698892 2874 flags.go:64] FLAG: --memory-manager-policy="None" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698895 2874 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698898 2874 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 18 16:53:46.707391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698901 2874 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698904 2874 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698910 2874 flags.go:64] FLAG: --node-status-max-images="50" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698913 2874 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698916 2874 flags.go:64] FLAG: --oom-score-adj="-999" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698919 2874 flags.go:64] FLAG: --pod-cidr="" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698922 2874 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698927 2874 flags.go:64] FLAG: --pod-manifest-path="" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698930 2874 flags.go:64] FLAG: --pod-max-pids="-1" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698933 2874 flags.go:64] FLAG: --pods-per-core="0" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698935 2874 flags.go:64] FLAG: --port="10250" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698938 2874 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698943 2874 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a0a51dad7a63c91c" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698946 2874 flags.go:64] FLAG: --qos-reserved="" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698949 2874 flags.go:64] FLAG: --read-only-port="10255" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698952 2874 flags.go:64] FLAG: --register-node="true" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698955 2874 flags.go:64] FLAG: --register-with-taints="" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698959 2874 flags.go:64] FLAG: --registry-burst="10" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698961 2874 flags.go:64] FLAG: --registry-qps="5" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698964 2874 flags.go:64] FLAG: --reserved-cpus="" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698967 2874 flags.go:64] FLAG: --reserved-memory="" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698970 2874 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698973 2874 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698976 2874 flags.go:64] FLAG: --rotate-certificates="false" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698979 2874 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 18 16:53:46.708015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698981 2874 flags.go:64] FLAG: --runonce="false" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698984 2874 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698987 2874 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698990 2874 flags.go:64] FLAG: --seccomp-default="false" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698993 2874 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698995 2874 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.698998 2874 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699001 2874 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699004 2874 flags.go:64] FLAG: --storage-driver-password="root" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699006 2874 flags.go:64] FLAG: --storage-driver-secure="false" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699009 2874 flags.go:64] FLAG: --storage-driver-table="stats" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699012 2874 flags.go:64] FLAG: --storage-driver-user="root" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699014 2874 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699017 2874 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699021 2874 flags.go:64] FLAG: --system-cgroups="" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699023 2874 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699029 2874 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699032 2874 flags.go:64] FLAG: --tls-cert-file="" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699034 2874 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699040 2874 flags.go:64] FLAG: --tls-min-version="" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699043 2874 flags.go:64] FLAG: --tls-private-key-file="" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699046 2874 flags.go:64] FLAG: --topology-manager-policy="none" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699049 2874 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699052 2874 flags.go:64] FLAG: --topology-manager-scope="container" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699054 2874 flags.go:64] FLAG: --v="2" Sep 18 16:53:46.708688 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699058 2874 flags.go:64] FLAG: --version="false" Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699062 2874 flags.go:64] FLAG: --vmodule="" Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699065 2874 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699068 2874 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699168 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699172 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699175 2874 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699178 2874 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699180 2874 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699183 2874 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699186 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699189 2874 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699191 2874 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699194 2874 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699197 2874 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699200 2874 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699202 2874 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699206 2874 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699209 2874 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699212 2874 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:53:46.709378 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699215 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699218 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699221 2874 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699224 2874 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699228 2874 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699231 2874 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699235 2874 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699238 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699241 2874 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699244 2874 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699246 2874 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699249 2874 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699252 2874 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699255 2874 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699258 2874 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699262 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699264 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699267 2874 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699270 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:53:46.710149 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699272 2874 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699275 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699277 2874 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699280 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699282 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699286 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699288 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699291 2874 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699293 2874 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699296 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699298 2874 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699301 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699303 2874 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699306 2874 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699309 2874 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699311 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699314 2874 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699316 2874 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699319 2874 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699323 2874 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:53:46.710773 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699325 2874 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699328 2874 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699330 2874 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699333 2874 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699335 2874 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699338 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699340 2874 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699343 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699346 2874 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699348 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699351 2874 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699354 2874 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699356 2874 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699359 2874 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699362 2874 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699364 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699367 2874 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699370 2874 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699373 2874 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699375 2874 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:53:46.711552 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699378 2874 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699381 2874 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699384 2874 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699386 2874 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699388 2874 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699391 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699394 2874 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699413 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699415 2874 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699418 2874 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699421 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699424 2874 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699427 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699429 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699432 2874 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699434 2874 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699437 2874 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699439 2874 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699441 2874 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699444 2874 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:53:46.712196 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699447 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699449 2874 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699452 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.699454 2874 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.699462 2874 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.705689 2874 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.705703 2874 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.705722 2874 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.705732 2874 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705786 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705792 2874 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705797 2874 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705800 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705803 2874 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:53:46.712776 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705806 2874 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705808 2874 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705811 2874 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705814 2874 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705816 2874 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705819 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705823 2874 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705827 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705830 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705832 2874 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705835 2874 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705838 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705841 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705844 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705846 2874 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705849 2874 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705852 2874 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705855 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705858 2874 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:53:46.713306 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705860 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705863 2874 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705866 2874 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705868 2874 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705872 2874 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705875 2874 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705878 2874 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705880 2874 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705883 2874 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705886 2874 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705888 2874 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705890 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705893 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705895 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705898 2874 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705900 2874 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705903 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705905 2874 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705908 2874 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705910 2874 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:53:46.714004 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705913 2874 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705915 2874 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705918 2874 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705920 2874 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705923 2874 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705926 2874 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705928 2874 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705931 2874 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705934 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705936 2874 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705939 2874 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705941 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705943 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705946 2874 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705949 2874 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705952 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705954 2874 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705957 2874 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705960 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705962 2874 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:53:46.714744 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705965 2874 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705967 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705970 2874 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705973 2874 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705975 2874 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705978 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705980 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705982 2874 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705985 2874 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705987 2874 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705990 2874 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705993 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705995 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.705999 2874 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706002 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706011 2874 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706014 2874 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706016 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706019 2874 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:53:46.715289 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706021 2874 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706024 2874 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706027 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706029 2874 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706031 2874 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706034 2874 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706037 2874 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706040 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706043 2874 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706046 2874 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706049 2874 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706052 2874 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706054 2874 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706057 2874 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706059 2874 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:53:46.715818 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706062 2874 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.706067 2874 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706181 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706187 2874 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706190 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706192 2874 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706195 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706198 2874 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706200 2874 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706203 2874 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706205 2874 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706207 2874 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706210 2874 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706212 2874 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706215 2874 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:53:46.716184 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706218 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706221 2874 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706223 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706225 2874 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706228 2874 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706232 2874 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706235 2874 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706238 2874 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706240 2874 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706243 2874 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706246 2874 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706249 2874 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706251 2874 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706254 2874 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706257 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706259 2874 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706261 2874 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706264 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706267 2874 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:53:46.716574 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706269 2874 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706272 2874 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706274 2874 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706277 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706279 2874 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706281 2874 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706284 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706286 2874 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706288 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706291 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706293 2874 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706296 2874 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706298 2874 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706301 2874 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706303 2874 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706306 2874 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706308 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706310 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706313 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706315 2874 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:53:46.717045 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706318 2874 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706320 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706323 2874 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706325 2874 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706328 2874 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706330 2874 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706333 2874 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706336 2874 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706338 2874 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706340 2874 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706343 2874 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706345 2874 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706348 2874 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706350 2874 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706353 2874 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706355 2874 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706357 2874 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706360 2874 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706362 2874 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706365 2874 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:53:46.717593 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706367 2874 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706370 2874 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706372 2874 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706374 2874 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706377 2874 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706379 2874 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706382 2874 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706384 2874 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706387 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706389 2874 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706391 2874 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706394 2874 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706410 2874 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706413 2874 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706415 2874 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706418 2874 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706428 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706431 2874 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706433 2874 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706436 2874 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:53:46.718081 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706440 2874 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:53:46.718707 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706443 2874 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:53:46.718707 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706446 2874 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:53:46.718707 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706449 2874 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:53:46.718707 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706451 2874 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:53:46.718707 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706454 2874 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:53:46.718707 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:53:46.706457 2874 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:53:46.718707 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.706463 2874 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:53:46.718707 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.706558 2874 server.go:962] "Client rotation is on, will bootstrap in background" Sep 18 16:53:46.718707 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.707010 2874 bootstrap.go:241] "Unhandled Error" err="unable to read existing bootstrap client config from /var/lib/kubelet/kubeconfig: invalid configuration: [unable to read client-cert /var/lib/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/kubelet/pki/kubelet-client-current.pem: no such file or directory, unable to read client-key /var/lib/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/kubelet/pki/kubelet-client-current.pem: no such file or directory]" logger="UnhandledError" Sep 18 16:53:46.718992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.707852 2874 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 18 16:53:46.718992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.707982 2874 server.go:1019] "Starting client certificate rotation" Sep 18 16:53:46.718992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.708096 2874 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:53:46.718992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.708126 2874 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:53:46.718992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.708426 2874 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 16:53:46.718992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.709423 2874 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 16:53:46.718992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.711379 2874 log.go:25] "Validated CRI v1 runtime API" Sep 18 16:53:46.718992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.712584 2874 log.go:25] "Validated CRI v1 image API" Sep 18 16:53:46.718992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.713756 2874 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 18 16:53:46.718992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.714590 2874 fs.go:135] Filesystem UUIDs: map[13d728e8-5990-4963-80c7-cc0f65acb881:/dev/nvme0n1p3 21b885ba-83e5-40d5-b1d8-0dcb82691ba1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 18 16:53:46.719249 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.714610 2874 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875/userdata/shm major:0 minor:45 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/759d5b14e1e91b6cee658f952c1e8b561710d1689920f53864344d64b06420c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/759d5b14e1e91b6cee658f952c1e8b561710d1689920f53864344d64b06420c2/userdata/shm major:0 minor:46 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/8e7b011f25ae689417116b710d7a1ed77a2910bb38fd1e5a2484f2d5d978183a/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ca469960855b39c9d49e506b50cf57bdc1bd7c4b0d629516e8b5f1883427ab19/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-55:{mountpoint:/var/lib/containers/storage/overlay/874cadcb9c55f57d5ad796efc65d3a03079dbee748bb205e72c9052e73c0ced3/merged major:0 minor:55 fsType:overlay blockSize:0}] Sep 18 16:53:46.719319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.718518 2874 manager.go:216] Machine: {Timestamp:2026-09-18 16:53:46.718008285 +0000 UTC m=+0.052372982 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108296 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d747d07f0c4236adc782c195ee27c SystemUUID:ec2d747d-07f0-c423-6adc-782c195ee27c BootID:734e3e40-5659-4eb5-98d8-080e884d706f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/759d5b14e1e91b6cee658f952c1e8b561710d1689920f53864344d64b06420c2/userdata/shm DeviceMajor:0 DeviceMinor:46 Capacity:67108864 Type:vfs Inodes:4005389 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-55 DeviceMajor:0 DeviceMinor:55 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875/userdata/shm DeviceMajor:0 DeviceMinor:45 Capacity:67108864 Type:vfs Inodes:4005389 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:36:35:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:36:35:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:98:37:c3:71:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 18 16:53:46.719319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.718617 2874 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 18 16:53:46.719319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.718653 2874 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 18 16:53:46.719319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.718885 2874 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 18 16:53:46.719319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.718908 2874 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 18 16:53:46.719319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719052 2874 topology_manager.go:138] "Creating topology manager with none policy" Sep 18 16:53:46.719319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719060 2874 container_manager_linux.go:309] "Creating device plugin manager" Sep 18 16:53:46.719319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719067 2874 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 16:53:46.719319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719176 2874 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 16:53:46.719319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719308 2874 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 18 16:53:46.719813 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719609 2874 state_mem.go:36] "Initialized new in-memory state store" Sep 18 16:53:46.719813 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719775 2874 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 18 16:53:46.719813 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719806 2874 kubelet.go:486] "Attempting to sync node with API server" Sep 18 16:53:46.719951 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719820 2874 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 18 16:53:46.719951 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719838 2874 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 18 16:53:46.719951 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719852 2874 kubelet.go:398] "Adding apiserver pod source" Sep 18 16:53:46.719951 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.719869 2874 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 18 16:53:46.720381 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.720362 2874 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Sep 18 16:53:46.720381 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.720380 2874 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 16:53:46.720563 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.720474 2874 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Sep 18 16:53:46.720563 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.720482 2874 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 16:53:46.721212 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.721164 2874 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 18 16:53:46.721873 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.721859 2874 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 18 16:53:46.721944 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.721888 2874 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 18 16:53:46.721981 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.721974 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 18 16:53:46.722016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.721985 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 18 16:53:46.722016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.721991 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 18 16:53:46.722016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.721995 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 18 16:53:46.722016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.722003 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 18 16:53:46.722016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.722009 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 18 16:53:46.722016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.722014 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 18 16:53:46.722016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.722018 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 18 16:53:46.722201 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.722025 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 18 16:53:46.722201 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.722031 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 18 16:53:46.722201 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.722038 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 18 16:53:46.722201 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.722043 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 18 16:53:46.722201 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.722068 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 18 16:53:46.722201 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.722073 2874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 18 16:53:46.723882 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.723857 2874 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:53:46.725166 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.725153 2874 server.go:1268] "Started kubelet" Sep 18 16:53:46.725554 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.725265 2874 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 18 16:53:46.725554 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.725238 2874 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 18 16:53:46.725554 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.725346 2874 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 18 16:53:46.726266 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.725748 2874 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 18 16:53:46.725998 ip-10-0-132-99 systemd[1]: Started Kubernetes Kubelet. Sep 18 16:53:46.727162 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.726524 2874 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 18 16:53:46.727162 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.726681 2874 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 18 16:53:46.727162 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.726695 2874 server.go:310] "Adding debug handlers to kubelet server" Sep 18 16:53:46.727162 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.726759 2874 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 18 16:53:46.727162 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.726769 2874 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 18 16:53:46.727162 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.726828 2874 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 18 16:53:46.727162 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.726837 2874 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:53:46.727162 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.726945 2874 reconstruct.go:97] "Volume reconstruction finished" Sep 18 16:53:46.727162 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.726954 2874 reconciler.go:29] "Reconciler: start to sync state" Sep 18 16:53:46.727614 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.727262 2874 factory.go:55] Registering systemd factory Sep 18 16:53:46.727614 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.727280 2874 factory.go:223] Registration of the systemd container factory successfully Sep 18 16:53:46.727736 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.727691 2874 factory.go:153] Registering CRI-O factory Sep 18 16:53:46.727788 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.727742 2874 factory.go:223] Registration of the crio container factory successfully Sep 18 16:53:46.727837 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.727812 2874 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 18 16:53:46.727837 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.727831 2874 factory.go:103] Registering Raw factory Sep 18 16:53:46.727925 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.727848 2874 manager.go:1196] Started watching for new ooms in manager Sep 18 16:53:46.728580 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.728565 2874 manager.go:349] Starting recovery of all containers Sep 18 16:53:46.729439 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.729416 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/5.log" Sep 18 16:53:46.729521 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.729467 2874 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 18 16:53:46.729894 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.729868 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:53:46.730001 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.729979 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 18 16:53:46.730060 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.730031 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:46.730131 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.730106 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:53:46.730761 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.730594 2874 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d58e17c1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.725128129 +0000 UTC m=+0.059492826,LastTimestamp:2026-09-18 16:53:46.725128129 +0000 UTC m=+0.059492826,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.730905 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.730873 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:53:46.732560 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.732276 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/5.log" Sep 18 16:53:46.741176 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.741126 2874 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 18 16:53:46.742620 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.742599 2874 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 18 16:53:46.742620 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.742623 2874 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 18 16:53:46.742751 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.742644 2874 kubelet.go:2448] "Starting kubelet main sync loop" Sep 18 16:53:46.742751 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.742714 2874 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Sep 18 16:53:46.744661 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.744641 2874 manager.go:354] Recovery completed Sep 18 16:53:46.745434 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.745392 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:53:46.745935 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.745919 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/5.log" Sep 18 16:53:46.746024 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.745943 2874 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="b915e7a912e31663470c39906dfe99e76383d8c39daada22ec99a66075623730" exitCode=1 Sep 18 16:53:46.749895 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.749881 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:46.751309 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.751288 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:46.751415 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.751323 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:46.751415 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.751335 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:46.751992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.751977 2874 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 18 16:53:46.751992 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.751991 2874 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 18 16:53:46.752110 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.751996 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:46.752110 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752007 2874 state_mem.go:36] "Initialized new in-memory state store" Sep 18 16:53:46.752179 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752158 2874 state_mem.go:88] "Updated default CPUSet" cpuSet="" Sep 18 16:53:46.752213 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752172 2874 state_mem.go:96] "Updated CPUSet assignments" assignments={} Sep 18 16:53:46.752213 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752188 2874 state_checkpoint.go:136] "State checkpoint: restored state from checkpoint" Sep 18 16:53:46.752213 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752194 2874 state_checkpoint.go:137] "State checkpoint: defaultCPUSet" defaultCpuSet="" Sep 18 16:53:46.752213 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752200 2874 policy_none.go:49] "None policy: Start" Sep 18 16:53:46.752213 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752208 2874 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 18 16:53:46.752358 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752218 2874 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 18 16:53:46.752358 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752310 2874 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Sep 18 16:53:46.752358 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752317 2874 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Sep 18 16:53:46.752358 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752322 2874 policy_none.go:47] "Start" Sep 18 16:53:46.752744 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752726 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:46.752821 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752770 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:46.752821 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.752787 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:46.755760 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.755745 2874 manager.go:337] "Starting Device Plugin manager" Sep 18 16:53:46.755849 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.755787 2874 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 18 16:53:46.755849 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.755799 2874 server.go:87] "Starting device plugin registration server" Sep 18 16:53:46.756080 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.756065 2874 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 18 16:53:46.756157 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.756083 2874 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 18 16:53:46.756214 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.756186 2874 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 18 16:53:46.756313 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.756296 2874 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 18 16:53:46.756383 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.756314 2874 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 18 16:53:46.756836 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.756817 2874 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 18 16:53:46.756931 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.756856 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:53:46.760307 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.760236 2874 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71d9511 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751309073 +0000 UTC m=+0.085673771,LastTimestamp:2026-09-18 16:53:46.751309073 +0000 UTC m=+0.085673771,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.768882 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.768807 2874 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71ddd33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751327539 +0000 UTC m=+0.085692236,LastTimestamp:2026-09-18 16:53:46.751327539 +0000 UTC m=+0.085692236,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.781288 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.781212 2874 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71e094d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751338829 +0000 UTC m=+0.085703527,LastTimestamp:2026-09-18 16:53:46.751338829 +0000 UTC m=+0.085703527,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.789846 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.789785 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71d9511\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71d9511 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751309073 +0000 UTC m=+0.085673771,LastTimestamp:2026-09-18 16:53:46.752744436 +0000 UTC m=+0.087109141,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.798915 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.798852 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71ddd33\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71ddd33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751327539 +0000 UTC m=+0.085692236,LastTimestamp:2026-09-18 16:53:46.75277875 +0000 UTC m=+0.087143453,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.806978 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.806924 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71e094d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71e094d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751338829 +0000 UTC m=+0.085703527,LastTimestamp:2026-09-18 16:53:46.7527936 +0000 UTC m=+0.087158305,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.814281 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.814222 2874 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d78e58d9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.758699225 +0000 UTC m=+0.093063914,LastTimestamp:2026-09-18 16:53:46.758699225 +0000 UTC m=+0.093063914,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.843736 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.843705 2874 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal"] Sep 18 16:53:46.843843 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.843833 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:46.844812 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.844795 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:46.844898 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.844831 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:46.844898 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.844842 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:46.845028 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.845018 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:46.845200 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.845179 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:46.846513 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.846489 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:46.846626 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.846541 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:46.846626 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.846564 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:46.846843 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.846808 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"b915e7a912e31663470c39906dfe99e76383d8c39daada22ec99a66075623730"} Sep 18 16:53:46.846924 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.846864 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875"} Sep 18 16:53:46.846924 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.846916 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" event={"ID":"b22dbf502d42c64e57bc1ca69e67914d","Type":"ContainerStarted","Data":"4a5018eb28bdd3e8d4978a764394e283e0c67411e973e9d1cffecba9770d19e6"} Sep 18 16:53:46.847073 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.846941 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" event={"ID":"b22dbf502d42c64e57bc1ca69e67914d","Type":"ContainerStarted","Data":"759d5b14e1e91b6cee658f952c1e8b561710d1689920f53864344d64b06420c2"} Sep 18 16:53:46.847073 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.847065 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:46.847326 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.847307 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:46.847389 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.847335 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:46.847389 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.847355 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:46.847591 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.847566 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:46.847970 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.847947 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:46.848039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.847978 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:46.848107 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.848089 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:46.848635 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.848613 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:46.853887 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.853817 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71d9511\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71d9511 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751309073 +0000 UTC m=+0.085673771,LastTimestamp:2026-09-18 16:53:46.844810659 +0000 UTC m=+0.179175356,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.856506 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.856493 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:46.857162 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.857124 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:46.857162 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.857156 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:46.857310 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.857173 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:46.857310 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.857198 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:46.864240 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.864169 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71ddd33\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71ddd33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751327539 +0000 UTC m=+0.085692236,LastTimestamp:2026-09-18 16:53:46.844835676 +0000 UTC m=+0.179200374,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.872581 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.872561 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:46.872665 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.872543 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71e094d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71e094d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751338829 +0000 UTC m=+0.085703527,LastTimestamp:2026-09-18 16:53:46.8448465 +0000 UTC m=+0.179211198,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.874965 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.874907 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71d9511\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71d9511 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751309073 +0000 UTC m=+0.085673771,LastTimestamp:2026-09-18 16:53:46.846511596 +0000 UTC m=+0.180876303,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.879765 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.879682 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71ddd33\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71ddd33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751327539 +0000 UTC m=+0.085692236,LastTimestamp:2026-09-18 16:53:46.846555694 +0000 UTC m=+0.180920396,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.883274 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.883216 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71e094d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71e094d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751338829 +0000 UTC m=+0.085703527,LastTimestamp:2026-09-18 16:53:46.846570084 +0000 UTC m=+0.180934784,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.890755 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.890696 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71d9511\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71d9511 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751309073 +0000 UTC m=+0.085673771,LastTimestamp:2026-09-18 16:53:46.847323401 +0000 UTC m=+0.181688102,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.898736 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.898685 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71ddd33\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71ddd33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751327539 +0000 UTC m=+0.085692236,LastTimestamp:2026-09-18 16:53:46.847341376 +0000 UTC m=+0.181706077,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.907536 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.907477 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71e094d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71e094d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751338829 +0000 UTC m=+0.085703527,LastTimestamp:2026-09-18 16:53:46.847361594 +0000 UTC m=+0.181726303,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.915924 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.915867 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71d9511\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71d9511 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751309073 +0000 UTC m=+0.085673771,LastTimestamp:2026-09-18 16:53:46.847966753 +0000 UTC m=+0.182331452,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.924933 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.924867 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71ddd33\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71ddd33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751327539 +0000 UTC m=+0.085692236,LastTimestamp:2026-09-18 16:53:46.847984189 +0000 UTC m=+0.182348890,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.927588 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.927568 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:53:46.927665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.927595 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:53:46.927665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:46.927611 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b22dbf502d42c64e57bc1ca69e67914d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.ec2.internal\" (UID: \"b22dbf502d42c64e57bc1ca69e67914d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" Sep 18 16:53:46.934568 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.934545 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Sep 18 16:53:46.934568 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.934508 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71e094d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71e094d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751338829 +0000 UTC m=+0.085703527,LastTimestamp:2026-09-18 16:53:46.848108575 +0000 UTC m=+0.182473278,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.938434 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.938354 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71d9511\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71d9511 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751309073 +0000 UTC m=+0.085673771,LastTimestamp:2026-09-18 16:53:46.85713901 +0000 UTC m=+0.191503713,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.943665 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.943602 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71ddd33\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71ddd33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751327539 +0000 UTC m=+0.085692236,LastTimestamp:2026-09-18 16:53:46.857164154 +0000 UTC m=+0.191528866,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:46.948233 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:46.948180 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71e094d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71e094d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751338829 +0000 UTC m=+0.085703527,LastTimestamp:2026-09-18 16:53:46.857178431 +0000 UTC m=+0.191543134,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:47.028415 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.028357 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.028542 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.028434 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.028542 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.028486 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.028542 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.028488 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c271c1096af51a1f9f8e05948cf36ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal\" (UID: \"8c271c1096af51a1f9f8e05948cf36ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.028542 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.028521 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b22dbf502d42c64e57bc1ca69e67914d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.ec2.internal\" (UID: \"b22dbf502d42c64e57bc1ca69e67914d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.028676 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.028480 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b22dbf502d42c64e57bc1ca69e67914d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-99.ec2.internal\" (UID: \"b22dbf502d42c64e57bc1ca69e67914d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.073529 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.073506 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:47.074510 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.074494 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:47.074580 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.074527 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:47.074580 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.074542 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:47.074580 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.074570 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.076562 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.076481 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71d9511\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71d9511 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751309073 +0000 UTC m=+0.085673771,LastTimestamp:2026-09-18 16:53:47.074508753 +0000 UTC m=+0.408873450,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:47.088707 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.088682 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.088761 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.088700 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-99.ec2.internal.18d67887d71ddd33\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-99.ec2.internal.18d67887d71ddd33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-99.ec2.internal,UID:ip-10-0-132-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:46.751327539 +0000 UTC m=+0.085692236,LastTimestamp:2026-09-18 16:53:47.074533237 +0000 UTC m=+0.408897936,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:47.160072 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.159998 2874 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887eed34632 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.149092402 +0000 UTC m=+0.483457087,LastTimestamp:2026-09-18 16:53:47.149092402 +0000 UTC m=+0.483457087,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:47.233146 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.233060 2874 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f35b9503 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.225134339 +0000 UTC m=+0.559499105,LastTimestamp:2026-09-18 16:53:47.225134339 +0000 UTC m=+0.559499105,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:47.241437 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.241349 2874 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f3dbfcff openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.233549567 +0000 UTC m=+0.567914265,LastTimestamp:2026-09-18 16:53:47.233549567 +0000 UTC m=+0.567914265,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:47.342997 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.342970 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Sep 18 16:53:47.489554 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.489495 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:47.490469 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.490449 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:47.490549 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.490485 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:47.490549 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.490495 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:47.490549 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.490520 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.505684 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.505657 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.576035 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.576008 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:53:47.737539 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.737510 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:47.750270 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.750209 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/5.log" Sep 18 16:53:47.750270 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.750256 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"87712fa1a14e03ced79e030d2ef2ea75b6832e723b3063f82ddcbf556582522f"} Sep 18 16:53:47.750395 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.750330 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:47.751848 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.751834 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:47.751896 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.751865 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:47.751896 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.751879 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:47.752108 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.752094 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:47.797686 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.797658 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:53:47.817850 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.817832 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:47.818718 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.818701 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:47.818787 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.818731 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:47.818787 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:47.818742 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:47.902391 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:47.902367 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:53:48.152822 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:48.152796 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Sep 18 16:53:48.294863 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:48.294824 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:53:48.305901 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.305878 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:48.307142 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.307124 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:48.307212 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.307154 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:48.307212 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.307188 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:48.307282 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.307216 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:48.327823 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:48.327792 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:48.740443 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.740416 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:48.751619 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.751597 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:48.752388 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.752372 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:48.752466 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.752416 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:48.752466 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.752429 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:48.752662 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:48.752650 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:48.882313 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:48.882277 2874 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:53:48.884940 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:48.884921 2874 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:53:49.642817 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:49.642782 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:53:49.744141 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:49.744113 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:49.761492 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:49.761458 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Sep 18 16:53:49.928212 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:49.928144 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:49.929128 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:49.929110 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:49.929225 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:49.929143 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:49.929225 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:49.929156 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:49.929225 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:49.929186 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:49.930801 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:49.930782 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:49.931541 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:49.931522 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:49.931619 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:49.931558 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:49.931619 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:49.931573 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:49.948660 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:49.948639 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:50.049994 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:50.049961 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:53:50.061860 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:50.061834 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:53:50.742095 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:50.742061 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:50.898002 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:50.897967 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:53:51.738760 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:51.738724 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:52.740250 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:52.740221 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:52.969969 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:52.969938 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s" Sep 18 16:53:53.149684 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:53.149654 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:53.150642 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:53.150618 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:53.150748 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:53.150660 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:53.150748 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:53.150674 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:53.150748 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:53.150702 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:53.171066 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:53.171045 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:53.228547 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:53.228526 2874 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:53:53.230887 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:53.230868 2874 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:53:53.651414 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:53.651377 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:53:53.739151 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:53.739127 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:53.822342 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:53.822313 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:53:54.171947 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:54.171921 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:54.172960 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:54.172944 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:54.173045 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:54.172974 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:54.173045 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:54.172984 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:54.738937 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:54.738911 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:55.375779 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:55.375745 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:53:55.728133 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:55.728100 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:53:55.738166 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:55.738143 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:56.740681 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:56.740653 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:56.757012 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:56.756990 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:53:57.740327 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:57.740292 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:58.738842 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.738813 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:58.767663 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.767638 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/6.log" Sep 18 16:53:58.768035 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.768020 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/5.log" Sep 18 16:53:58.768099 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.768055 2874 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="87712fa1a14e03ced79e030d2ef2ea75b6832e723b3063f82ddcbf556582522f" exitCode=1 Sep 18 16:53:58.768099 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.768083 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"87712fa1a14e03ced79e030d2ef2ea75b6832e723b3063f82ddcbf556582522f"} Sep 18 16:53:58.768204 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.768165 2874 scope.go:117] "RemoveContainer" containerID="b915e7a912e31663470c39906dfe99e76383d8c39daada22ec99a66075623730" Sep 18 16:53:58.768256 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.768244 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:58.769600 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.769231 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:58.769600 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.769259 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:58.769600 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.769272 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:58.769600 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:58.769491 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:58.769600 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:58.769547 2874 scope.go:117] "RemoveContainer" containerID="b915e7a912e31663470c39906dfe99e76383d8c39daada22ec99a66075623730" Sep 18 16:53:58.778306 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:58.778274 2874 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_5 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: 'b915e7a912e31663470c39906dfe99e76383d8c39daada22ec99a66075623730'" containerID="b915e7a912e31663470c39906dfe99e76383d8c39daada22ec99a66075623730" Sep 18 16:53:58.778382 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:58.778327 2874 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_5 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: 'b915e7a912e31663470c39906dfe99e76383d8c39daada22ec99a66075623730'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" logger="UnhandledError" Sep 18 16:53:58.778521 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:58.778502 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:53:58.789678 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:58.789595 2874 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:58.778460237 +0000 UTC m=+12.112824938,LastTimestamp:2026-09-18 16:53:58.778460237 +0000 UTC m=+12.112824938,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:53:59.380797 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:59.380761 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:53:59.572081 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:59.572049 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:53:59.573072 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:59.573053 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:53:59.573184 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:59.573093 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:53:59.573184 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:59.573107 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:53:59.573184 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:59.573136 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:59.589790 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:53:59.589758 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:53:59.739169 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:59.739097 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:53:59.770732 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:53:59.770704 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/6.log" Sep 18 16:54:00.739393 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:00.739362 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:01.740243 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:01.740211 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:01.990996 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:01.990898 2874 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:54:01.994064 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:01.994041 2874 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:54:02.168715 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:02.168686 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:54:02.211046 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:02.211019 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:02.211905 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:02.211884 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:02.211986 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:02.211919 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:02.211986 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:02.211936 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:02.740895 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:02.740868 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:03.068366 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:03.068285 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:54:03.739717 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:03.739690 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:04.135046 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:04.135018 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:54:04.740659 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:04.740628 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:05.022881 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:05.022797 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:54:05.739957 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:05.739928 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:06.389050 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:06.389018 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:54:06.590199 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:06.590155 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:06.591186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:06.591169 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:06.591252 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:06.591201 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:06.591252 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:06.591211 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:06.591252 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:06.591238 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:06.610879 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:06.610859 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:06.740993 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:06.740929 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:06.757222 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:06.757205 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:54:07.739048 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:07.739018 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:08.740595 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:08.740565 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:09.738853 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:09.738819 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:09.743173 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:09.743157 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:09.744022 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:09.744007 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:09.744082 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:09.744036 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:09.744082 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:09.744047 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:09.744242 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:09.744231 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:09.753958 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:09.753880 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887eed34632\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887eed34632 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.149092402 +0000 UTC m=+0.483457087,LastTimestamp:2026-09-18 16:54:09.744950141 +0000 UTC m=+23.079314845,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:54:09.833351 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:09.833282 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f35b9503\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f35b9503 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.225134339 +0000 UTC m=+0.559499105,LastTimestamp:2026-09-18 16:54:09.822723517 +0000 UTC m=+23.157088220,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:54:09.840856 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:09.840788 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f3dbfcff\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f3dbfcff openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.233549567 +0000 UTC m=+0.567914265,LastTimestamp:2026-09-18 16:54:09.831368123 +0000 UTC m=+23.165732831,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:54:10.740490 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:10.740462 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:10.787653 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:10.787631 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/6.log" Sep 18 16:54:10.787981 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:10.787672 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"b51827ca5bc52bb6fa8dc71ce134b900144967aafd5334ba705b8eb0cd2a1383"} Sep 18 16:54:10.787981 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:10.787776 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:10.788525 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:10.788507 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:10.788591 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:10.788538 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:10.788591 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:10.788547 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:10.788724 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:10.788712 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:11.742017 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:11.741992 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:12.740633 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:12.740600 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:13.397610 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:13.397574 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:54:13.611550 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:13.611510 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:13.612566 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:13.612540 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:13.612664 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:13.612578 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:13.612664 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:13.612592 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:13.612664 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:13.612620 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:13.630373 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:13.630340 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:13.740042 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:13.739970 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:14.741102 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:14.741075 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:15.739022 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:15.738995 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:16.742587 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:16.742559 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:16.757942 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:16.757919 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:54:17.740206 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:17.740177 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:17.995332 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:17.995238 2874 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:54:17.998086 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:17.998063 2874 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:54:17.998143 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:17.998090 2874 certificate_manager.go:461] "Reached backoff limit, still unable to rotate certs" err="timed out waiting for the condition" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:54:18.737921 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:18.737894 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:19.731800 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:19.731772 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:19.732231 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:19.731819 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:54:20.407938 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:20.407901 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:54:20.631123 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:20.631085 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:20.632122 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:20.632103 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:20.632185 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:20.632141 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:20.632185 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:20.632158 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:20.632262 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:20.632190 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:20.649479 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:20.649454 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:20.732874 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:20.732809 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:20.733698 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:20.733681 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:20.733813 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:20.733715 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:20.733813 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:20.733729 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:20.740799 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:20.740777 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:21.743504 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.743474 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:21.803814 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.803788 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/7.log" Sep 18 16:54:21.804227 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.804213 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/6.log" Sep 18 16:54:21.804275 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.804246 2874 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="b51827ca5bc52bb6fa8dc71ce134b900144967aafd5334ba705b8eb0cd2a1383" exitCode=1 Sep 18 16:54:21.804321 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.804272 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"b51827ca5bc52bb6fa8dc71ce134b900144967aafd5334ba705b8eb0cd2a1383"} Sep 18 16:54:21.804374 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.804341 2874 scope.go:117] "RemoveContainer" containerID="87712fa1a14e03ced79e030d2ef2ea75b6832e723b3063f82ddcbf556582522f" Sep 18 16:54:21.804466 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.804453 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:21.805629 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.805570 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:21.805629 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.805597 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:21.805629 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.805609 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:21.805816 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:21.805803 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:21.805865 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:21.805855 2874 scope.go:117] "RemoveContainer" containerID="87712fa1a14e03ced79e030d2ef2ea75b6832e723b3063f82ddcbf556582522f" Sep 18 16:54:21.814465 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:21.814439 2874 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_6 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: '87712fa1a14e03ced79e030d2ef2ea75b6832e723b3063f82ddcbf556582522f'" containerID="87712fa1a14e03ced79e030d2ef2ea75b6832e723b3063f82ddcbf556582522f" Sep 18 16:54:21.814553 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:21.814485 2874 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_6 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: '87712fa1a14e03ced79e030d2ef2ea75b6832e723b3063f82ddcbf556582522f'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" logger="UnhandledError" Sep 18 16:54:21.814658 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:21.814639 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:54:21.825580 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:21.825504 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:58.778460237 +0000 UTC m=+12.112824938,LastTimestamp:2026-09-18 16:54:21.814602934 +0000 UTC m=+35.148967642,Count:2,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:54:22.742393 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:22.742362 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:22.807194 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:22.807171 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/7.log" Sep 18 16:54:23.039838 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:23.039768 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:54:23.740901 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:23.740868 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:24.742271 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:24.742238 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:25.737984 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:25.737948 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:25.769489 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:25.769450 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:54:26.364846 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:26.364809 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:54:26.740260 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:26.740227 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:26.758763 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:26.758738 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:54:27.419186 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:27.419151 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:54:27.649851 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:27.649818 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:27.651266 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:27.651248 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:27.651356 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:27.651307 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:27.651356 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:27.651321 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:27.651356 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:27.651346 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:27.668351 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:27.668326 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:27.740578 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:27.740512 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:28.741020 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:28.740989 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:29.739979 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:29.739952 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:30.739768 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:30.739738 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:31.739022 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:31.738989 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:32.739775 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:32.739744 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:33.742425 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:33.742385 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:34.428300 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:34.428270 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:54:34.669201 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:34.669164 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:34.670180 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:34.670160 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:34.670275 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:34.670195 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:34.670275 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:34.670205 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:34.670275 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:34.670231 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:34.688559 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:34.688502 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:34.738750 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:34.738720 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:35.739792 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:35.739760 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:35.743153 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:35.743138 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:35.743988 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:35.743969 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:35.744091 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:35.744003 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:35.744091 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:35.744017 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:35.744224 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:35.744209 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:35.744416 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:35.744382 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:54:35.755034 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:35.754966 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:58.778460237 +0000 UTC m=+12.112824938,LastTimestamp:2026-09-18 16:54:35.744350584 +0000 UTC m=+49.078715288,Count:3,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:54:36.739101 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:36.739070 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:36.759783 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:36.759763 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:54:37.738994 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:37.738965 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:38.743659 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:38.743626 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:39.739476 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:39.739448 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:40.739548 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:40.739514 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:41.440664 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:41.440631 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:54:41.689553 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:41.689517 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:41.690549 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:41.690529 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:41.690634 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:41.690562 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:41.690634 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:41.690572 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:41.690634 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:41.690603 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:41.707367 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:41.707310 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:41.740017 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:41.739993 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:42.738582 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:42.738556 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:43.739383 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:43.739353 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:44.741184 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:44.741146 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:45.741473 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:45.741448 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:46.741215 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:46.741191 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:46.760638 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:46.760609 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:54:47.738644 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:47.738621 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:48.449685 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:48.449652 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:54:48.707797 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:48.707722 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:48.709730 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:48.709709 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:48.709805 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:48.709751 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:48.709805 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:48.709793 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:48.709877 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:48.709823 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:48.726983 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:48.726947 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:48.740116 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:48.740088 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:49.739802 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:49.739767 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:49.999864 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:49.999786 2874 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:54:50.002662 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:50.002641 2874 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:54:50.738898 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.738872 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:50.743340 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.743321 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:50.744152 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.744134 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:50.744223 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.744168 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:50.744223 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.744178 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:50.744388 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:50.744376 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:50.754186 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:50.754107 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887eed34632\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887eed34632 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.149092402 +0000 UTC m=+0.483457087,LastTimestamp:2026-09-18 16:54:50.745086363 +0000 UTC m=+64.079451053,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:54:50.825753 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:50.825682 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f35b9503\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f35b9503 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.225134339 +0000 UTC m=+0.559499105,LastTimestamp:2026-09-18 16:54:50.81518141 +0000 UTC m=+64.149546116,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:54:50.838922 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:50.838855 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f3dbfcff\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f3dbfcff openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.233549567 +0000 UTC m=+0.567914265,LastTimestamp:2026-09-18 16:54:50.823772265 +0000 UTC m=+64.158136972,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:54:50.848582 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.848568 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/7.log" Sep 18 16:54:50.848660 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.848607 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"c256129e0d79e39169117d2ad99f1d626185826ea15218dc6cb1de87b2da274a"} Sep 18 16:54:50.848714 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.848704 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:50.849537 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.849522 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:50.849594 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.849550 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:50.849594 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:50.849560 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:50.849749 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:50.849733 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:51.739926 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:51.739899 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:52.740095 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:52.740064 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:53.083014 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:53.082922 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:53.085246 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:53.085228 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:53.085301 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:53.085260 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:53.085301 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:53.085271 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:53.739808 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:53.739781 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:54.742021 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:54.741989 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:55.460188 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:55.460151 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:54:55.727691 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:55.727631 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:54:55.728738 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:55.728716 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:54:55.728832 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:55.728758 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:54:55.728832 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:55.728773 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:54:55.728832 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:55.728804 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:55.738172 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:55.738151 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:55.742132 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:55.742113 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:54:56.732357 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:56.732328 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:56.760832 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:54:56.760807 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:54:57.739700 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:57.739668 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:58.740348 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:58.740316 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:54:59.739741 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:54:59.739704 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:00.742494 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:00.742465 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:01.742211 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:01.742182 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:02.469008 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:02.468976 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:55:02.741857 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.741792 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:02.742829 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.742811 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:02.743790 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.743774 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:02.743881 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.743802 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:02.743881 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.743813 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:02.743881 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.743836 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:02.763004 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:02.762981 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:02.867297 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.867274 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/8.log" Sep 18 16:55:02.867683 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.867663 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/7.log" Sep 18 16:55:02.867750 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.867696 2874 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="c256129e0d79e39169117d2ad99f1d626185826ea15218dc6cb1de87b2da274a" exitCode=1 Sep 18 16:55:02.867750 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.867720 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"c256129e0d79e39169117d2ad99f1d626185826ea15218dc6cb1de87b2da274a"} Sep 18 16:55:02.867814 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.867758 2874 scope.go:117] "RemoveContainer" containerID="b51827ca5bc52bb6fa8dc71ce134b900144967aafd5334ba705b8eb0cd2a1383" Sep 18 16:55:02.867846 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.867830 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:02.868634 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.868613 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:02.868749 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.868650 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:02.868749 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.868666 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:02.869071 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:02.868936 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:02.869071 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:02.868998 2874 scope.go:117] "RemoveContainer" containerID="b51827ca5bc52bb6fa8dc71ce134b900144967aafd5334ba705b8eb0cd2a1383" Sep 18 16:55:02.877964 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:02.877939 2874 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_7 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: 'b51827ca5bc52bb6fa8dc71ce134b900144967aafd5334ba705b8eb0cd2a1383'" containerID="b51827ca5bc52bb6fa8dc71ce134b900144967aafd5334ba705b8eb0cd2a1383" Sep 18 16:55:02.878066 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:02.877983 2874 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_7 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: 'b51827ca5bc52bb6fa8dc71ce134b900144967aafd5334ba705b8eb0cd2a1383'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" logger="UnhandledError" Sep 18 16:55:02.878159 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:02.878141 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:55:02.887475 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:02.887380 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:58.778460237 +0000 UTC m=+12.112824938,LastTimestamp:2026-09-18 16:55:02.87810022 +0000 UTC m=+76.212464928,Count:4,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:55:03.739934 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:03.739906 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:03.870560 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:03.870533 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/8.log" Sep 18 16:55:04.387048 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:04.387013 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:55:04.686603 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:04.686529 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:55:04.741032 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:04.741005 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:05.739237 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:05.739210 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:06.432602 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:06.432567 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:55:06.740472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:06.740386 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:06.743576 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:06.743556 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:06.744476 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:06.744447 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:06.744574 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:06.744494 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:06.744574 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:06.744509 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:06.744696 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:06.744689 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:06.761876 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:06.761854 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:55:07.738565 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:07.738538 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:08.737749 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:08.737715 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:09.485392 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:09.485351 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:55:09.737982 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:09.737914 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:09.763468 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:09.763443 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:09.764371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:09.764354 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:09.764464 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:09.764384 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:09.764464 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:09.764394 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:09.764464 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:09.764447 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:09.783726 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:09.783701 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:10.738376 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:10.738347 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:11.740174 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:11.740141 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:12.741311 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:12.741285 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:13.737943 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:13.737912 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:14.741324 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:14.741296 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:15.741859 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:15.741832 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:16.495049 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:16.495015 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:55:16.708074 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:16.708042 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:55:16.741518 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:16.741492 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:16.743628 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:16.743607 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:16.744441 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:16.744425 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:16.744506 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:16.744458 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:16.744506 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:16.744470 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:16.744643 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:16.744631 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:16.744782 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:16.744767 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:55:16.757112 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:16.757002 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:58.778460237 +0000 UTC m=+12.112824938,LastTimestamp:2026-09-18 16:55:16.744739204 +0000 UTC m=+90.079103908,Count:5,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:55:16.762130 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:16.762114 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:55:16.784815 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:16.784790 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:16.785591 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:16.785575 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:16.785656 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:16.785604 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:16.785656 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:16.785614 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:16.785656 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:16.785640 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:16.804081 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:16.804058 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:17.738100 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:17.738073 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:18.741480 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:18.741452 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:19.741325 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:19.741298 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:20.740191 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:20.740167 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:21.739186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:21.739154 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:21.999946 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:21.999844 2874 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:55:22.002539 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:22.002514 2874 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:55:22.741554 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:22.741532 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:23.504689 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:23.504654 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:55:23.740022 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:23.739985 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:23.804557 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:23.804497 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:23.805480 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:23.805461 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:23.805552 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:23.805493 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:23.805552 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:23.805502 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:23.805552 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:23.805529 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:23.822278 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:23.822252 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:24.739075 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:24.739047 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:25.740654 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:25.740630 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:26.738185 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:26.738161 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:26.762497 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:26.762473 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:55:27.739671 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:27.739640 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:28.739513 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:28.739479 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:29.739063 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:29.739025 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:30.513996 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:30.513955 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:55:30.741130 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:30.741098 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:30.823095 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:30.823015 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:30.824079 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:30.824058 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:30.824205 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:30.824090 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:30.824205 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:30.824100 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:30.824205 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:30.824125 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:30.840911 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:30.840881 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:31.740500 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:31.740466 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:31.743858 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:31.743840 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:31.744707 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:31.744689 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:31.744841 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:31.744719 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:31.744841 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:31.744729 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:31.744941 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:31.744930 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:31.745080 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:31.745065 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:55:31.755166 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:31.755096 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:58.778460237 +0000 UTC m=+12.112824938,LastTimestamp:2026-09-18 16:55:31.74503813 +0000 UTC m=+105.079402829,Count:6,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:55:32.738665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:32.738630 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:33.741044 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:33.741011 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:34.739778 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:34.739744 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:35.739853 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:35.739817 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:36.739299 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:36.739267 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:36.762834 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:36.762812 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:55:37.523047 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:37.523010 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:55:37.739854 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:37.739825 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:37.841808 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:37.841733 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:37.843058 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:37.843040 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:37.843164 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:37.843070 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:37.843164 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:37.843080 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:37.843164 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:37.843106 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:37.860771 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:37.860748 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:38.739965 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:38.739936 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:39.740120 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:39.740085 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:40.739489 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:40.739459 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:41.740663 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:41.740630 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:42.740900 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:42.740872 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:43.738527 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:43.738497 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:44.533005 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:44.532967 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:55:44.740447 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.740412 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:44.743670 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.743655 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:44.744566 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.744547 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:44.744678 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.744577 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:44.744678 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.744589 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:44.744793 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:44.744782 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:44.753648 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:44.753549 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887eed34632\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887eed34632 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.149092402 +0000 UTC m=+0.483457087,LastTimestamp:2026-09-18 16:55:44.7455147 +0000 UTC m=+118.079879385,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:55:44.830885 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:44.830813 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f35b9503\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f35b9503 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.225134339 +0000 UTC m=+0.559499105,LastTimestamp:2026-09-18 16:55:44.821937683 +0000 UTC m=+118.156302390,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:55:44.840951 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:44.840887 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f3dbfcff\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d67887f3dbfcff openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:47.233549567 +0000 UTC m=+0.567914265,LastTimestamp:2026-09-18 16:55:44.830575937 +0000 UTC m=+118.164940622,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:55:44.861022 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.861003 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:44.861846 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.861833 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:44.861907 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.861862 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:44.861907 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.861875 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:44.861907 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.861901 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:44.881264 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:44.881240 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:44.933468 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.933435 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/8.log" Sep 18 16:55:44.933622 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.933491 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"651c540d09203e2f13cf25ddc0be8e5e06bfc1ec0568fba943b0d72ea4c5b093"} Sep 18 16:55:44.933622 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.933595 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:44.934359 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.934343 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:44.934422 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.934376 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:44.934422 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:44.934387 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:44.934697 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:44.934685 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:45.742844 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:45.742814 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:46.746979 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:46.746948 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:46.763895 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:46.763867 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:55:47.740088 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:47.740057 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:48.747884 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:48.747856 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:49.744131 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:49.744102 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:50.739388 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:50.739362 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:51.154803 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:51.154771 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:55:51.542956 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:51.542871 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:55:51.740081 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:51.740051 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:51.881856 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:51.881831 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:51.883592 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:51.883574 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:51.883712 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:51.883604 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:51.883712 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:51.883614 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:51.883712 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:51.883643 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:51.904654 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:51.904625 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:52.743185 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:52.743153 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:53.740540 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:53.740513 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:53.999541 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:53.999455 2874 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:55:54.002395 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:54.002376 2874 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError" Sep 18 16:55:54.744191 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:54.744166 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:55.181831 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:55.181794 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:55:55.742724 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.742692 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:55.951456 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.951430 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/9.log" Sep 18 16:55:55.951858 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.951843 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/8.log" Sep 18 16:55:55.951913 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.951875 2874 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="651c540d09203e2f13cf25ddc0be8e5e06bfc1ec0568fba943b0d72ea4c5b093" exitCode=1 Sep 18 16:55:55.951913 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.951903 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"651c540d09203e2f13cf25ddc0be8e5e06bfc1ec0568fba943b0d72ea4c5b093"} Sep 18 16:55:55.952000 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.951980 2874 scope.go:117] "RemoveContainer" containerID="c256129e0d79e39169117d2ad99f1d626185826ea15218dc6cb1de87b2da274a" Sep 18 16:55:55.952098 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.952048 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:55.958637 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.958619 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:55.958718 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.958656 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:55.958718 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.958675 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:55.958900 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:55.958885 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:55.958985 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:55.958938 2874 scope.go:117] "RemoveContainer" containerID="c256129e0d79e39169117d2ad99f1d626185826ea15218dc6cb1de87b2da274a" Sep 18 16:55:55.964354 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:55.964326 2874 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_8 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: 'c256129e0d79e39169117d2ad99f1d626185826ea15218dc6cb1de87b2da274a'" containerID="c256129e0d79e39169117d2ad99f1d626185826ea15218dc6cb1de87b2da274a" Sep 18 16:55:55.964454 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:55.964369 2874 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_8 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: 'c256129e0d79e39169117d2ad99f1d626185826ea15218dc6cb1de87b2da274a'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" logger="UnhandledError" Sep 18 16:55:55.964533 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:55.964515 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:55:55.973187 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:55.973115 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:58.778460237 +0000 UTC m=+12.112824938,LastTimestamp:2026-09-18 16:55:55.964480974 +0000 UTC m=+129.298845665,Count:7,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:55:56.739792 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:56.739765 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:56.764024 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:56.764001 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:55:56.954981 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:56.954954 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/9.log" Sep 18 16:55:57.741290 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:57.741259 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:58.556743 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:58.556705 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:55:58.739201 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:58.739172 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:55:58.905310 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:58.905281 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:55:58.906379 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:58.906357 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:55:58.906521 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:58.906413 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:55:58.906521 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:58.906431 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:55:58.906521 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:58.906476 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:58.925017 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:58.924991 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:55:59.031738 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:55:59.031702 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:55:59.743108 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:55:59.743083 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:00.739367 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:00.739335 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:00.877199 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:00.877168 2874 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 18 16:56:01.746619 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:01.746589 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:02.739117 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:02.739088 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:03.740829 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:03.740804 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:04.392019 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:04.391973 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:04.393005 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:04.392986 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:04.393113 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:04.393021 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:04.393113 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:04.393035 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:04.742205 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:04.742146 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:05.564915 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:05.564889 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:56:05.741122 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:05.741100 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:05.926099 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:05.926073 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:05.927062 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:05.927040 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:05.927151 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:05.927077 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:05.927151 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:05.927088 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:05.927151 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:05.927120 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:05.947241 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:05.947219 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:06.740416 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:06.740372 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:06.764658 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:06.764625 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:07.740954 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:07.740922 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:08.738853 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:08.738819 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:08.743214 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:08.743200 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:08.744505 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:08.744488 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:08.744560 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:08.744522 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:08.744560 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:08.744536 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:08.744757 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:08.744744 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:08.744942 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:08.744926 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:56:08.756135 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:08.756065 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:58.778460237 +0000 UTC m=+12.112824938,LastTimestamp:2026-09-18 16:56:08.744888237 +0000 UTC m=+142.079252946,Count:8,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:56:09.741804 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:09.741776 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:10.740391 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:10.740360 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:11.741124 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:11.741098 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:12.578225 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:12.578195 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:56:12.738541 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:12.738516 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:12.947507 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:12.947467 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:12.948483 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:12.948466 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:12.948556 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:12.948499 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:12.948556 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:12.948510 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:12.948556 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:12.948537 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:12.973190 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:12.973166 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:13.739485 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:13.739457 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:14.743964 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:14.743931 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:15.739971 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:15.739939 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:16.743320 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:16.743293 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:16.765656 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:16.765623 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:17.740251 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:17.740221 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:18.742093 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:18.742065 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:19.586365 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:19.586333 2874 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Sep 18 16:56:19.745026 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:19.745000 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:19.973614 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:19.973583 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:19.974677 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:19.974660 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:19.974810 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:19.974689 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:19.974810 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:19.974701 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:19.974810 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:19.974726 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:19.999762 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:19.999741 2874 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:20.740814 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:20.740784 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:21.738608 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:21.738581 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:21.742941 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:21.742927 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:21.743014 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:21.742928 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:21.744575 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:21.744558 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:21.744682 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:21.744595 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:21.744682 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:21.744611 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:21.744682 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:21.744560 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:21.744682 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:21.744659 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:21.744682 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:21.744669 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:21.744934 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:21.744847 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:21.744934 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:21.744909 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:21.745041 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:21.745025 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:56:21.753170 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:21.753098 2874 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal.18d6788aa3fd584d openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal,UID:8c271c1096af51a1f9f8e05948cf36ce,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce),Source:EventSource{Component:kubelet,Host:ip-10-0-132-99.ec2.internal,},FirstTimestamp:2026-09-18 16:53:58.778460237 +0000 UTC m=+12.112824938,LastTimestamp:2026-09-18 16:56:21.745003601 +0000 UTC m=+155.079368286,Count:9,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-99.ec2.internal,}" Sep 18 16:56:22.739530 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:22.739501 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:23.746268 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:23.746236 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:24.743185 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:24.743152 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:25.743977 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:25.743951 2874 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:56:25.999952 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:25.999888 2874 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:56:26.027867 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:26.027836 2874 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 16:56:26.155397 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:26.155365 2874 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rn6zn" Sep 18 16:56:26.185515 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:26.185491 2874 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rn6zn" Sep 18 16:56:26.609957 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:26.609928 2874 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:26.708621 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:26.708589 2874 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 18 16:56:26.756904 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:26.756876 2874 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-132-99.ec2.internal" not found Sep 18 16:56:26.766063 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:26.766039 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:26.781916 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:26.781900 2874 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-132-99.ec2.internal" not found Sep 18 16:56:26.848342 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:26.848316 2874 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-132-99.ec2.internal" not found Sep 18 16:56:27.000842 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:27.000821 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:27.001690 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:27.001671 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:27.001795 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:27.001701 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:27.001795 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:27.001712 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:27.001795 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:27.001737 2874 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:27.017123 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:27.017101 2874 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:27.017123 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:27.017123 2874 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-99.ec2.internal\": node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:27.038717 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:27.038697 2874 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:27.139061 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:27.139041 2874 kubelet_node_status.go:448] "Node not becoming ready in time after startup" Sep 18 16:56:27.189096 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:27.189052 2874 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:51:26 +0000 UTC" deadline="2028-03-03 01:43:58.344727891 +0000 UTC" Sep 18 16:56:27.189096 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:27.189091 2874 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12752h47m31.155639957s" Sep 18 16:56:28.072937 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:28.072787 2874 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:56:31.787226 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:31.787189 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:56:33.743706 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:33.743665 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:33.745777 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:33.745751 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:33.745913 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:33.745784 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:33.745913 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:33.745795 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:33.746035 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:33.745980 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:33.746133 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:33.746115 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:56:36.766494 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:36.766463 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:36.787895 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:36.787862 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:56:37.435088 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:37.435043 2874 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-99.ec2.internal\": node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:41.788674 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:41.788637 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:56:44.743016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:44.742975 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:44.744002 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:44.743983 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:44.744103 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:44.744016 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:44.744103 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:44.744028 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:44.744248 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:44.744236 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:44.744395 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:44.744380 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:56:46.766897 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:46.766866 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:46.789888 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:46.789860 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:56:47.732070 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:47.732033 2874 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-99.ec2.internal\": node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:49.689099 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:49.689072 2874 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:56:51.790945 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:51.790909 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:56:55.214666 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:55.214632 2874 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:56:56.767042 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:56.766996 2874 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:56.791451 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:56.791418 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:56:57.743686 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:57.743642 2874 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:56:57.745275 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:57.745255 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:56:57.745430 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:57.745286 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:56:57.745430 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:57.745295 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:56:57.745522 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:57.745516 2874 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-99.ec2.internal\" not found" node="ip-10-0-132-99.ec2.internal" Sep 18 16:56:57.745665 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:57.745649 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:56:57.782040 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:57.782016 2874 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-99.ec2.internal\": node \"ip-10-0-132-99.ec2.internal\" not found" Sep 18 16:56:58.684445 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.684418 2874 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:56:58.727659 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.727629 2874 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" Sep 18 16:56:58.738187 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.738169 2874 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 16:56:58.739588 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.739576 2874 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" Sep 18 16:56:58.751660 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.751644 2874 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 16:56:58.831043 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.831018 2874 apiserver.go:52] "Watching apiserver" Sep 18 16:56:58.838540 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.838518 2874 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 18 16:56:58.838851 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.838827 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-4xfr7","kube-system/konnectivity-agent-wd6bk","kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal","openshift-cluster-node-tuning-operator/tuned-fn6d2","kube-system/global-pull-secret-syncer-t4fzq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal","openshift-multus/multus-2hbpw","openshift-multus/multus-additional-cni-plugins-bl7jm","openshift-multus/network-metrics-daemon-mqrhp"] Sep 18 16:56:58.841374 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.841351 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:56:58.841713 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.841698 2874 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 18 16:56:58.845262 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.845216 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.847917 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.847898 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:56:58.848016 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:58.847987 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:56:58.850154 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.850134 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:58.852328 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.852309 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.854093 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.854079 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 18 16:56:58.854414 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.854389 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:58.854986 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.854964 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 18 16:56:58.854986 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.854977 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 18 16:56:58.855220 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.854987 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 18 16:56:58.855220 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.854977 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:56:58.855220 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.855051 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 18 16:56:58.855220 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.854989 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pxctw\"" Sep 18 16:56:58.855220 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.855095 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 18 16:56:58.855220 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.854972 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 18 16:56:58.855500 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.855237 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 18 16:56:58.855500 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.855371 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7q7k4\"" Sep 18 16:56:58.855500 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.855451 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 18 16:56:58.855500 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.855460 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l249t\"" Sep 18 16:56:58.855500 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.855489 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pvsd6\"" Sep 18 16:56:58.855655 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.855513 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 18 16:56:58.856649 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.856632 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:56:58.856735 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:58.856713 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:56:58.857149 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.857134 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 18 16:56:58.857801 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.857781 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 18 16:56:58.857899 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.857842 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bjmf8\"" Sep 18 16:56:58.858926 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.858906 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.859091 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:58.859061 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:56:58.862306 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.862287 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 18 16:56:58.862612 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.862315 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 18 16:56:58.862612 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.862434 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 18 16:56:58.862612 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.862598 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-brbn9\"" Sep 18 16:56:58.862797 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.862661 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 18 16:56:58.862852 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.862837 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 18 16:56:58.862883 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.862871 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 18 16:56:58.874987 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.874971 2874 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 16:56:58.921146 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.921119 2874 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sbvdf" Sep 18 16:56:58.922288 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.922253 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-99.ec2.internal" podStartSLOduration=0.922236942 podStartE2EDuration="922.236942ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:56:58.922203129 +0000 UTC m=+192.256567835" watchObservedRunningTime="2026-09-18 16:56:58.922236942 +0000 UTC m=+192.256601649" Sep 18 16:56:58.927714 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.927700 2874 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 18 16:56:58.935852 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.935803 2874 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sbvdf" Sep 18 16:56:58.991200 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991175 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-var-lib-kubelet\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.991353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991205 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-cni-dir\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.991353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991225 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-daemon-config\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.991353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991241 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-bin\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.991353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991280 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-netd\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.991353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991316 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-env-overrides\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.991353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991336 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-sys\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991359 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrwt\" (UniqueName: \"kubernetes.io/projected/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-kube-api-access-gjrwt\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991376 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-hostroot\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991391 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh9s\" (UniqueName: \"kubernetes.io/projected/bb0c72b0-0d55-4136-99ce-d50036c55d91-kube-api-access-phh9s\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991422 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb442e64-9a04-459c-814a-71a5bf49c5f7-kubelet-config\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991448 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb442e64-9a04-459c-814a-71a5bf49c5f7-dbus\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991463 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991476 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-socket-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991492 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-registration-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991512 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz7ch\" (UniqueName: \"kubernetes.io/projected/16a22de8-89ae-401d-b3ff-8554161ebd3d-kube-api-access-jz7ch\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991555 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-device-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991580 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-sys-fs\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:58.991639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991604 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-socket-dir-parent\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991646 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-run-k8s-cni-cncf-io\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991684 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-openvswitch\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991712 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-sysconfig\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991743 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-host\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991766 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6kw\" (UniqueName: \"kubernetes.io/projected/b825e74d-e9fc-43d1-91d5-c318fe1dab02-kube-api-access-bp6kw\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991790 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-systemd\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991811 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-script-lib\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991833 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-lib-modules\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991857 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991892 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-slash\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991923 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-config\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991946 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c4d9103-40fb-4c3e-b830-bc31eb5edd90-konnectivity-ca\") pod \"konnectivity-agent-wd6bk\" (UID: \"3c4d9103-40fb-4c3e-b830-bc31eb5edd90\") " pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991971 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.991995 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-cnibin\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992019 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovn-node-metrics-cert\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.992192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992039 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-tmp\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992062 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992085 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvj8\" (UniqueName: \"kubernetes.io/projected/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-kube-api-access-8jvj8\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992107 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992122 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-kubernetes\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992137 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-conf-dir\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992161 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16a22de8-89ae-401d-b3ff-8554161ebd3d-cni-binary-copy\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992197 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-run-multus-certs\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992222 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992247 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-run-netns\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992268 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-netns\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992291 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-system-cni-dir\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992313 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992335 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-var-lib-cni-bin\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992360 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-etc-openvswitch\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992415 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-sysctl-d\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.992890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992438 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-var-lib-openvswitch\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992470 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-node-log\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992499 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-log-socket\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992521 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-modprobe-d\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992543 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-tuned\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992567 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992589 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-var-lib-kubelet\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992613 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-system-cni-dir\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992639 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-run\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992664 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992680 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992697 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-var-lib-cni-multus\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992710 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-etc-kubernetes\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992723 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-kubelet\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992738 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-ovn\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992760 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c4d9103-40fb-4c3e-b830-bc31eb5edd90-agent-certs\") pod \"konnectivity-agent-wd6bk\" (UID: \"3c4d9103-40fb-4c3e-b830-bc31eb5edd90\") " pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992779 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-cnibin\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:58.993353 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992794 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-systemd-units\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.993844 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992809 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlc7m\" (UniqueName: \"kubernetes.io/projected/3a5603d8-009a-4758-ad7b-421e5da3a9f3-kube-api-access-dlc7m\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:58.993844 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992838 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-sysctl-conf\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:58.993844 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:58.992861 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-systemd\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.093557 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.093527 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-slash\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.093677 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.093565 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-config\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.093677 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.093581 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c4d9103-40fb-4c3e-b830-bc31eb5edd90-konnectivity-ca\") pod \"konnectivity-agent-wd6bk\" (UID: \"3c4d9103-40fb-4c3e-b830-bc31eb5edd90\") " pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:56:59.093677 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.093596 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.093677 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.093627 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-slash\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.093860 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.093711 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-cnibin\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094045 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovn-node-metrics-cert\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094098 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-tmp\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094145 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094169 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-cnibin\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094178 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvj8\" (UniqueName: \"kubernetes.io/projected/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-kube-api-access-8jvj8\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094216 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094245 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094249 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-kubernetes\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094436 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-conf-dir\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094483 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16a22de8-89ae-401d-b3ff-8554161ebd3d-cni-binary-copy\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094504 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-config\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094511 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094529 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-run-multus-certs\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094520 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c4d9103-40fb-4c3e-b830-bc31eb5edd90-konnectivity-ca\") pod \"konnectivity-agent-wd6bk\" (UID: \"3c4d9103-40fb-4c3e-b830-bc31eb5edd90\") " pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094557 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094663 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-run-netns\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.095807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094733 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-kubernetes\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094693 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-netns\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094784 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094791 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-system-cni-dir\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094797 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-conf-dir\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094918 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-run-netns\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094951 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.094981 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-system-cni-dir\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095055 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-run-multus-certs\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095072 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-var-lib-cni-bin\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095113 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-netns\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095146 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-etc-openvswitch\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:59.095241 2874 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095256 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-sysctl-d\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095326 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-var-lib-openvswitch\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095494 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-node-log\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095514 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16a22de8-89ae-401d-b3ff-8554161ebd3d-cni-binary-copy\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095525 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-log-socket\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.096665 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095424 2874 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095553 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-modprobe-d\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095616 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-tuned\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095647 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-sysctl-d\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095712 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-etc-openvswitch\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095737 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-var-lib-cni-bin\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095753 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-node-log\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095760 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095830 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095847 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-var-lib-openvswitch\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095894 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-modprobe-d\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095908 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-var-lib-kubelet\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095943 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-log-socket\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.095988 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-system-cni-dir\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096047 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-var-lib-kubelet\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096056 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-system-cni-dir\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096377 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-run\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.097472 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096447 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-run\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:59.096452 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs podName:bb0c72b0-0d55-4136-99ce-d50036c55d91 nodeName:}" failed. No retries permitted until 2026-09-18 16:56:59.59637953 +0000 UTC m=+192.930744267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs") pod "network-metrics-daemon-mqrhp" (UID: "bb0c72b0-0d55-4136-99ce-d50036c55d91") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096489 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096505 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096535 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096560 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-var-lib-cni-multus\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096570 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096588 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-etc-kubernetes\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096668 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-kubelet\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096694 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-ovn\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096700 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096722 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c4d9103-40fb-4c3e-b830-bc31eb5edd90-agent-certs\") pod \"konnectivity-agent-wd6bk\" (UID: \"3c4d9103-40fb-4c3e-b830-bc31eb5edd90\") " pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096746 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-etc-kubernetes\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096746 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-cnibin\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096783 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-systemd-units\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096857 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-var-lib-cni-multus\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096893 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-ovn\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.098303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.096906 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-kubelet\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097016 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc7m\" (UniqueName: \"kubernetes.io/projected/3a5603d8-009a-4758-ad7b-421e5da3a9f3-kube-api-access-dlc7m\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097154 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-systemd-units\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097619 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-sysctl-conf\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097668 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-systemd\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097704 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-var-lib-kubelet\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097733 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-cni-dir\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097757 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-daemon-config\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097775 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-sysctl-conf\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097785 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-bin\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097838 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-bin\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097846 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-netd\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097897 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-systemd\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097897 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-env-overrides\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097934 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-sys\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097965 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrwt\" (UniqueName: \"kubernetes.io/projected/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-kube-api-access-gjrwt\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.097996 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-hostroot\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098033 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh9s\" (UniqueName: \"kubernetes.io/projected/bb0c72b0-0d55-4136-99ce-d50036c55d91-kube-api-access-phh9s\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:56:59.099186 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098060 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb442e64-9a04-459c-814a-71a5bf49c5f7-kubelet-config\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098088 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb442e64-9a04-459c-814a-71a5bf49c5f7-dbus\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098120 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098149 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-socket-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098234 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-netd\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098365 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-socket-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098422 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-env-overrides\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098456 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-hostroot\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098555 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb442e64-9a04-459c-814a-71a5bf49c5f7-kubelet-config\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098621 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-cni-dir\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098673 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-registration-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098761 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz7ch\" (UniqueName: \"kubernetes.io/projected/16a22de8-89ae-401d-b3ff-8554161ebd3d-kube-api-access-jz7ch\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098761 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-registration-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098866 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-sys\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.098910 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-var-lib-kubelet\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:59.098585 2874 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099020 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-device-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099215 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb442e64-9a04-459c-814a-71a5bf49c5f7-dbus\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:56:59.099995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099748 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-sys-fs\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099790 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-socket-dir-parent\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099815 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-run-k8s-cni-cncf-io\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099840 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-openvswitch\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099873 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-sysconfig\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099883 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-tmp\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099896 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-host\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099902 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-sys-fs\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099918 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-tuned\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099920 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6kw\" (UniqueName: \"kubernetes.io/projected/b825e74d-e9fc-43d1-91d5-c318fe1dab02-kube-api-access-bp6kw\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099956 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b825e74d-e9fc-43d1-91d5-c318fe1dab02-device-dir\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099978 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-host\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099978 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-host-run-k8s-cni-cncf-io\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.099986 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-systemd\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100009 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-openvswitch\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:59.100016 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret podName:cb442e64-9a04-459c-814a-71a5bf49c5f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:56:59.599997764 +0000 UTC m=+192.934362462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret") pod "global-pull-secret-syncer-t4fzq" (UID: "cb442e64-9a04-459c-814a-71a5bf49c5f7") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100022 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-systemd\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.100726 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100037 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-script-lib\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100048 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovn-node-metrics-cert\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100064 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-lib-modules\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100091 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100179 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-socket-dir-parent\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100233 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-etc-sysconfig\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100261 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-lib-modules\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100327 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c4d9103-40fb-4c3e-b830-bc31eb5edd90-agent-certs\") pod \"konnectivity-agent-wd6bk\" (UID: \"3c4d9103-40fb-4c3e-b830-bc31eb5edd90\") " pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100341 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100364 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-cnibin\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100367 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/16a22de8-89ae-401d-b3ff-8554161ebd3d-multus-daemon-config\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.101192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.100547 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-script-lib\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.127212 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.127183 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh9s\" (UniqueName: \"kubernetes.io/projected/bb0c72b0-0d55-4136-99ce-d50036c55d91-kube-api-access-phh9s\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:56:59.127897 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.127863 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrwt\" (UniqueName: \"kubernetes.io/projected/d8254aa7-70d3-4bdc-87a4-457d77cda2e5-kube-api-access-gjrwt\") pod \"tuned-fn6d2\" (UID: \"d8254aa7-70d3-4bdc-87a4-457d77cda2e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.128017 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.127929 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvj8\" (UniqueName: \"kubernetes.io/projected/e9465c98-ea01-4cd6-83b7-485bfb6b54f0-kube-api-access-8jvj8\") pod \"multus-additional-cni-plugins-bl7jm\" (UID: \"e9465c98-ea01-4cd6-83b7-485bfb6b54f0\") " pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.128289 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.128270 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlc7m\" (UniqueName: \"kubernetes.io/projected/3a5603d8-009a-4758-ad7b-421e5da3a9f3-kube-api-access-dlc7m\") pod \"ovnkube-node-4xfr7\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.133571 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.133550 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz7ch\" (UniqueName: \"kubernetes.io/projected/16a22de8-89ae-401d-b3ff-8554161ebd3d-kube-api-access-jz7ch\") pod \"multus-2hbpw\" (UID: \"16a22de8-89ae-401d-b3ff-8554161ebd3d\") " pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.133717 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.133701 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6kw\" (UniqueName: \"kubernetes.io/projected/b825e74d-e9fc-43d1-91d5-c318fe1dab02-kube-api-access-bp6kw\") pod \"aws-ebs-csi-driver-node-ldv6s\" (UID: \"b825e74d-e9fc-43d1-91d5-c318fe1dab02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.151650 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.151627 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:56:59.156977 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.156959 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" Sep 18 16:56:59.161740 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:56:59.161717 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4d9103_40fb_4c3e_b830_bc31eb5edd90.slice/crio-78462073eb4ea0f1e04ff37b07a2eeab276a3da57e4ff95d2349588b9f7900ee WatchSource:0}: Error finding container 78462073eb4ea0f1e04ff37b07a2eeab276a3da57e4ff95d2349588b9f7900ee: Status 404 returned error can't find the container with id 78462073eb4ea0f1e04ff37b07a2eeab276a3da57e4ff95d2349588b9f7900ee Sep 18 16:56:59.162650 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.162633 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" Sep 18 16:56:59.163287 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.163271 2874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:56:59.165857 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:56:59.165838 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8254aa7_70d3_4bdc_87a4_457d77cda2e5.slice/crio-ad3a9a21b66a9fe2a95ac7ee4549fdf2b087f181e2a36532f24baf5f424da015 WatchSource:0}: Error finding container ad3a9a21b66a9fe2a95ac7ee4549fdf2b087f181e2a36532f24baf5f424da015: Status 404 returned error can't find the container with id ad3a9a21b66a9fe2a95ac7ee4549fdf2b087f181e2a36532f24baf5f424da015 Sep 18 16:56:59.169050 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.169033 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2hbpw" Sep 18 16:56:59.170724 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:56:59.170704 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb825e74d_e9fc_43d1_91d5_c318fe1dab02.slice/crio-634cc9e0c9d4e06320db978eef36bf9dc4c9dadf872ed265f3159bf60ae2a553 WatchSource:0}: Error finding container 634cc9e0c9d4e06320db978eef36bf9dc4c9dadf872ed265f3159bf60ae2a553: Status 404 returned error can't find the container with id 634cc9e0c9d4e06320db978eef36bf9dc4c9dadf872ed265f3159bf60ae2a553 Sep 18 16:56:59.174704 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.174687 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" Sep 18 16:56:59.178198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.178183 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:56:59.178354 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:56:59.178337 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a22de8_89ae_401d_b3ff_8554161ebd3d.slice/crio-046672a6735b5458d9b1ff46c9184caab49d53138cef9ebc5d5299c5f2f991fe WatchSource:0}: Error finding container 046672a6735b5458d9b1ff46c9184caab49d53138cef9ebc5d5299c5f2f991fe: Status 404 returned error can't find the container with id 046672a6735b5458d9b1ff46c9184caab49d53138cef9ebc5d5299c5f2f991fe Sep 18 16:56:59.184479 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:56:59.184457 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9465c98_ea01_4cd6_83b7_485bfb6b54f0.slice/crio-90a7fb00567d207e19cf825c5d4079be77b69cbd98744281e6812e87555df35f WatchSource:0}: Error finding container 90a7fb00567d207e19cf825c5d4079be77b69cbd98744281e6812e87555df35f: Status 404 returned error can't find the container with id 90a7fb00567d207e19cf825c5d4079be77b69cbd98744281e6812e87555df35f Sep 18 16:56:59.187953 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:56:59.187933 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5603d8_009a_4758_ad7b_421e5da3a9f3.slice/crio-aee3d358ab3c285eec71e2cfab0cc6ebb979e815da31cac34d5d216f748c7b50 WatchSource:0}: Error finding container aee3d358ab3c285eec71e2cfab0cc6ebb979e815da31cac34d5d216f748c7b50: Status 404 returned error can't find the container with id aee3d358ab3c285eec71e2cfab0cc6ebb979e815da31cac34d5d216f748c7b50 Sep 18 16:56:59.602874 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.602584 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:56:59.602874 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.602648 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:56:59.602874 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:59.602722 2874 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:56:59.602874 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:59.602755 2874 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:56:59.602874 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:59.602798 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs podName:bb0c72b0-0d55-4136-99ce-d50036c55d91 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:00.602776606 +0000 UTC m=+193.937141310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs") pod "network-metrics-daemon-mqrhp" (UID: "bb0c72b0-0d55-4136-99ce-d50036c55d91") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:56:59.602874 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:56:59.602820 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret podName:cb442e64-9a04-459c-814a-71a5bf49c5f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:00.602808452 +0000 UTC m=+193.937173151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret") pod "global-pull-secret-syncer-t4fzq" (UID: "cb442e64-9a04-459c-814a-71a5bf49c5f7") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:56:59.938440 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.938380 2874 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:51:58 +0000 UTC" deadline="2028-05-17 11:33:01.179613406 +0000 UTC" Sep 18 16:56:59.938440 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:56:59.938438 2874 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14562h36m1.241180708s" Sep 18 16:57:00.050128 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.050087 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2hbpw" event={"ID":"16a22de8-89ae-401d-b3ff-8554161ebd3d","Type":"ContainerStarted","Data":"046672a6735b5458d9b1ff46c9184caab49d53138cef9ebc5d5299c5f2f991fe"} Sep 18 16:57:00.053386 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.053355 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" event={"ID":"b825e74d-e9fc-43d1-91d5-c318fe1dab02","Type":"ContainerStarted","Data":"634cc9e0c9d4e06320db978eef36bf9dc4c9dadf872ed265f3159bf60ae2a553"} Sep 18 16:57:00.060621 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.060560 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" event={"ID":"d8254aa7-70d3-4bdc-87a4-457d77cda2e5","Type":"ContainerStarted","Data":"ad3a9a21b66a9fe2a95ac7ee4549fdf2b087f181e2a36532f24baf5f424da015"} Sep 18 16:57:00.062588 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.062189 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wd6bk" event={"ID":"3c4d9103-40fb-4c3e-b830-bc31eb5edd90","Type":"ContainerStarted","Data":"78462073eb4ea0f1e04ff37b07a2eeab276a3da57e4ff95d2349588b9f7900ee"} Sep 18 16:57:00.064515 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.064445 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerStarted","Data":"aee3d358ab3c285eec71e2cfab0cc6ebb979e815da31cac34d5d216f748c7b50"} Sep 18 16:57:00.066583 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.066559 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" event={"ID":"e9465c98-ea01-4cd6-83b7-485bfb6b54f0","Type":"ContainerStarted","Data":"90a7fb00567d207e19cf825c5d4079be77b69cbd98744281e6812e87555df35f"} Sep 18 16:57:00.617230 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.616540 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:00.617230 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.616608 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:00.617230 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:00.616725 2874 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:57:00.617230 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:00.616786 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret podName:cb442e64-9a04-459c-814a-71a5bf49c5f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:02.61676639 +0000 UTC m=+195.951131078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret") pod "global-pull-secret-syncer-t4fzq" (UID: "cb442e64-9a04-459c-814a-71a5bf49c5f7") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:57:00.617230 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:00.617148 2874 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:57:00.617230 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:00.617192 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs podName:bb0c72b0-0d55-4136-99ce-d50036c55d91 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:02.617178954 +0000 UTC m=+195.951543654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs") pod "network-metrics-daemon-mqrhp" (UID: "bb0c72b0-0d55-4136-99ce-d50036c55d91") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:57:00.746136 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.745460 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:00.746136 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:00.745597 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:00.746136 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.745972 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:00.746136 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:00.746080 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:00.938645 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.938599 2874 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:51:58 +0000 UTC" deadline="2028-06-14 14:10:39.377011536 +0000 UTC" Sep 18 16:57:00.938645 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:00.938635 2874 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15237h13m38.438380527s" Sep 18 16:57:01.363487 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:01.363124 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-npjd7"] Sep 18 16:57:01.366984 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:01.366597 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:01.366984 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:01.366668 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:01.523683 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:01.523643 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dcrn\" (UniqueName: \"kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn\") pod \"network-check-target-npjd7\" (UID: \"e9379c30-e23f-4635-a9c1-73d866893686\") " pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:01.625343 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:01.624902 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dcrn\" (UniqueName: \"kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn\") pod \"network-check-target-npjd7\" (UID: \"e9379c30-e23f-4635-a9c1-73d866893686\") " pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:01.634479 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:01.634440 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:57:01.634479 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:01.634483 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:57:01.634710 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:01.634499 2874 projected.go:196] Error preparing data for projected volume kube-api-access-9dcrn for pod openshift-network-diagnostics/network-check-target-npjd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:01.634710 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:01.634565 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn podName:e9379c30-e23f-4635-a9c1-73d866893686 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:02.134546058 +0000 UTC m=+195.468910754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9dcrn" (UniqueName: "kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn") pod "network-check-target-npjd7" (UID: "e9379c30-e23f-4635-a9c1-73d866893686") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:01.793308 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:01.793034 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:57:02.229736 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:02.229693 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dcrn\" (UniqueName: \"kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn\") pod \"network-check-target-npjd7\" (UID: \"e9379c30-e23f-4635-a9c1-73d866893686\") " pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:02.230188 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.229889 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:57:02.230188 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.229918 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:57:02.230188 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.229932 2874 projected.go:196] Error preparing data for projected volume kube-api-access-9dcrn for pod openshift-network-diagnostics/network-check-target-npjd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:02.230188 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.229995 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn podName:e9379c30-e23f-4635-a9c1-73d866893686 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:03.229974122 +0000 UTC m=+196.564338825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dcrn" (UniqueName: "kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn") pod "network-check-target-npjd7" (UID: "e9379c30-e23f-4635-a9c1-73d866893686") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:02.633418 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:02.633364 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:02.633567 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:02.633467 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:02.633627 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.633609 2874 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:57:02.633677 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.633669 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs podName:bb0c72b0-0d55-4136-99ce-d50036c55d91 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:06.63365241 +0000 UTC m=+199.968017110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs") pod "network-metrics-daemon-mqrhp" (UID: "bb0c72b0-0d55-4136-99ce-d50036c55d91") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:57:02.634060 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.634046 2874 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:57:02.634105 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.634096 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret podName:cb442e64-9a04-459c-814a-71a5bf49c5f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:06.634082083 +0000 UTC m=+199.968446780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret") pod "global-pull-secret-syncer-t4fzq" (UID: "cb442e64-9a04-459c-814a-71a5bf49c5f7") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:57:02.743250 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:02.743005 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:02.743250 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:02.743016 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:02.743250 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.743135 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:02.743250 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:02.743168 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:02.743689 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.743264 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:02.743689 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:02.743351 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:03.185145 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.184281 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-br879"] Sep 18 16:57:03.188214 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.187647 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:03.193770 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.193333 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8pbqv\"" Sep 18 16:57:03.195615 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.195587 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 18 16:57:03.196233 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.195836 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 18 16:57:03.196233 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.196064 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 18 16:57:03.237457 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.237422 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dcrn\" (UniqueName: \"kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn\") pod \"network-check-target-npjd7\" (UID: \"e9379c30-e23f-4635-a9c1-73d866893686\") " pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:03.238001 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:03.237586 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:57:03.238001 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:03.237608 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:57:03.238001 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:03.237621 2874 projected.go:196] Error preparing data for projected volume kube-api-access-9dcrn for pod openshift-network-diagnostics/network-check-target-npjd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:03.238001 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:03.237687 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn podName:e9379c30-e23f-4635-a9c1-73d866893686 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:05.237668433 +0000 UTC m=+198.572033123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dcrn" (UniqueName: "kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn") pod "network-check-target-npjd7" (UID: "e9379c30-e23f-4635-a9c1-73d866893686") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:03.338600 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.338557 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb9022da-0935-4c13-9fdc-102da03b843e-host\") pod \"node-ca-br879\" (UID: \"eb9022da-0935-4c13-9fdc-102da03b843e\") " pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:03.338784 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.338609 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb9022da-0935-4c13-9fdc-102da03b843e-serviceca\") pod \"node-ca-br879\" (UID: \"eb9022da-0935-4c13-9fdc-102da03b843e\") " pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:03.338784 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.338636 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkqr\" (UniqueName: \"kubernetes.io/projected/eb9022da-0935-4c13-9fdc-102da03b843e-kube-api-access-kgkqr\") pod \"node-ca-br879\" (UID: \"eb9022da-0935-4c13-9fdc-102da03b843e\") " pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:03.439075 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.438935 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb9022da-0935-4c13-9fdc-102da03b843e-host\") pod \"node-ca-br879\" (UID: \"eb9022da-0935-4c13-9fdc-102da03b843e\") " pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:03.439075 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.438969 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb9022da-0935-4c13-9fdc-102da03b843e-serviceca\") pod \"node-ca-br879\" (UID: \"eb9022da-0935-4c13-9fdc-102da03b843e\") " pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:03.439075 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.438985 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkqr\" (UniqueName: \"kubernetes.io/projected/eb9022da-0935-4c13-9fdc-102da03b843e-kube-api-access-kgkqr\") pod \"node-ca-br879\" (UID: \"eb9022da-0935-4c13-9fdc-102da03b843e\") " pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:03.439075 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.439063 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb9022da-0935-4c13-9fdc-102da03b843e-host\") pod \"node-ca-br879\" (UID: \"eb9022da-0935-4c13-9fdc-102da03b843e\") " pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:03.439570 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.439543 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb9022da-0935-4c13-9fdc-102da03b843e-serviceca\") pod \"node-ca-br879\" (UID: \"eb9022da-0935-4c13-9fdc-102da03b843e\") " pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:03.488601 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.488567 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkqr\" (UniqueName: \"kubernetes.io/projected/eb9022da-0935-4c13-9fdc-102da03b843e-kube-api-access-kgkqr\") pod \"node-ca-br879\" (UID: \"eb9022da-0935-4c13-9fdc-102da03b843e\") " pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:03.502347 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:03.502308 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-br879" Sep 18 16:57:04.749314 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.746706 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:04.749314 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:04.746833 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:04.749314 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.747208 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:04.749314 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:04.747306 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:04.749314 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.747373 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:04.749314 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:04.747479 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:04.749314 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.748655 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-nn99k"] Sep 18 16:57:04.754712 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.754690 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:04.757668 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.757595 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:57:04.757668 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.757636 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 18 16:57:04.757668 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.757644 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nq4f2\"" Sep 18 16:57:04.758828 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.758639 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 18 16:57:04.851935 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.851817 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16cad111-725f-4c32-92e9-114a389ea4c2-host-slash\") pod \"iptables-alerter-nn99k\" (UID: \"16cad111-725f-4c32-92e9-114a389ea4c2\") " pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:04.851935 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.851910 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/16cad111-725f-4c32-92e9-114a389ea4c2-iptables-alerter-script\") pod \"iptables-alerter-nn99k\" (UID: \"16cad111-725f-4c32-92e9-114a389ea4c2\") " pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:04.851935 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.851943 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn8l7\" (UniqueName: \"kubernetes.io/projected/16cad111-725f-4c32-92e9-114a389ea4c2-kube-api-access-fn8l7\") pod \"iptables-alerter-nn99k\" (UID: \"16cad111-725f-4c32-92e9-114a389ea4c2\") " pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:04.952712 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.952621 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16cad111-725f-4c32-92e9-114a389ea4c2-host-slash\") pod \"iptables-alerter-nn99k\" (UID: \"16cad111-725f-4c32-92e9-114a389ea4c2\") " pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:04.952712 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.952705 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/16cad111-725f-4c32-92e9-114a389ea4c2-iptables-alerter-script\") pod \"iptables-alerter-nn99k\" (UID: \"16cad111-725f-4c32-92e9-114a389ea4c2\") " pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:04.952930 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.952739 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn8l7\" (UniqueName: \"kubernetes.io/projected/16cad111-725f-4c32-92e9-114a389ea4c2-kube-api-access-fn8l7\") pod \"iptables-alerter-nn99k\" (UID: \"16cad111-725f-4c32-92e9-114a389ea4c2\") " pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:04.952930 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.952801 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16cad111-725f-4c32-92e9-114a389ea4c2-host-slash\") pod \"iptables-alerter-nn99k\" (UID: \"16cad111-725f-4c32-92e9-114a389ea4c2\") " pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:04.953481 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.953450 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/16cad111-725f-4c32-92e9-114a389ea4c2-iptables-alerter-script\") pod \"iptables-alerter-nn99k\" (UID: \"16cad111-725f-4c32-92e9-114a389ea4c2\") " pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:04.967374 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:04.967339 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn8l7\" (UniqueName: \"kubernetes.io/projected/16cad111-725f-4c32-92e9-114a389ea4c2-kube-api-access-fn8l7\") pod \"iptables-alerter-nn99k\" (UID: \"16cad111-725f-4c32-92e9-114a389ea4c2\") " pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:05.067044 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:05.067005 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nn99k" Sep 18 16:57:05.255592 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:05.254887 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dcrn\" (UniqueName: \"kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn\") pod \"network-check-target-npjd7\" (UID: \"e9379c30-e23f-4635-a9c1-73d866893686\") " pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:05.255592 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:05.255098 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:57:05.255592 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:05.255122 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:57:05.255592 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:05.255135 2874 projected.go:196] Error preparing data for projected volume kube-api-access-9dcrn for pod openshift-network-diagnostics/network-check-target-npjd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:05.255592 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:05.255192 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn podName:e9379c30-e23f-4635-a9c1-73d866893686 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:09.255174654 +0000 UTC m=+202.589539343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dcrn" (UniqueName: "kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn") pod "network-check-target-npjd7" (UID: "e9379c30-e23f-4635-a9c1-73d866893686") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:06.663452 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:06.663412 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:06.663903 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:06.663490 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:06.663903 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:06.663532 2874 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:57:06.663903 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:06.663609 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs podName:bb0c72b0-0d55-4136-99ce-d50036c55d91 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:14.663586024 +0000 UTC m=+207.997950716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs") pod "network-metrics-daemon-mqrhp" (UID: "bb0c72b0-0d55-4136-99ce-d50036c55d91") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:57:06.663903 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:06.663639 2874 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:57:06.663903 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:06.663698 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret podName:cb442e64-9a04-459c-814a-71a5bf49c5f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:14.663680392 +0000 UTC m=+207.998045080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret") pod "global-pull-secret-syncer-t4fzq" (UID: "cb442e64-9a04-459c-814a-71a5bf49c5f7") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:57:06.744581 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:06.744537 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:06.744755 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:06.744633 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:06.744755 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:06.744655 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:06.744845 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:06.744808 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:06.744899 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:06.744868 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:06.744964 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:06.744946 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:06.794463 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:06.794380 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:57:08.743912 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:08.743877 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:08.743912 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:08.743911 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:08.744557 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:08.743882 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:08.744557 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:08.744015 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:08.744557 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:08.744089 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:08.744557 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:08.744178 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:09.281193 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:09.281144 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dcrn\" (UniqueName: \"kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn\") pod \"network-check-target-npjd7\" (UID: \"e9379c30-e23f-4635-a9c1-73d866893686\") " pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:09.281369 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:09.281351 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:57:09.281458 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:09.281379 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:57:09.281458 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:09.281389 2874 projected.go:196] Error preparing data for projected volume kube-api-access-9dcrn for pod openshift-network-diagnostics/network-check-target-npjd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:09.281558 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:09.281471 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn podName:e9379c30-e23f-4635-a9c1-73d866893686 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:17.281454187 +0000 UTC m=+210.615818876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dcrn" (UniqueName: "kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn") pod "network-check-target-npjd7" (UID: "e9379c30-e23f-4635-a9c1-73d866893686") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:09.743736 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:09.743684 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podUID="8c271c1096af51a1f9f8e05948cf36ce" Sep 18 16:57:10.743594 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:10.743512 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:10.744025 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:10.743632 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:10.744025 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:10.743644 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:10.744025 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:10.743751 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:10.744025 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:10.743814 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:10.744025 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:10.743888 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:11.795442 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:11.795384 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:57:12.742878 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:12.742836 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:12.742878 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:12.742871 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:12.743130 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:12.742846 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:12.743130 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:12.742974 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:12.743130 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:12.743049 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:12.743268 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:12.743134 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:14.219710 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:14.219685 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9022da_0935_4c13_9fdc_102da03b843e.slice/crio-19ee56105d111386d0ed8a38303b3c4d86b72a873bfb3d270e56cf3bf3a830bc WatchSource:0}: Error finding container 19ee56105d111386d0ed8a38303b3c4d86b72a873bfb3d270e56cf3bf3a830bc: Status 404 returned error can't find the container with id 19ee56105d111386d0ed8a38303b3c4d86b72a873bfb3d270e56cf3bf3a830bc Sep 18 16:57:14.276163 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:14.276121 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cad111_725f_4c32_92e9_114a389ea4c2.slice/crio-f8a7b6f6c571efade64b7d1a21359ae5696cf97f39c94be94e5ee1eb8157037c WatchSource:0}: Error finding container f8a7b6f6c571efade64b7d1a21359ae5696cf97f39c94be94e5ee1eb8157037c: Status 404 returned error can't find the container with id f8a7b6f6c571efade64b7d1a21359ae5696cf97f39c94be94e5ee1eb8157037c Sep 18 16:57:14.710327 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:14.710100 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:14.710483 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:14.710380 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:14.710483 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:14.710238 2874 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:57:14.710576 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:14.710487 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs podName:bb0c72b0-0d55-4136-99ce-d50036c55d91 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:30.710465991 +0000 UTC m=+224.044830689 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs") pod "network-metrics-daemon-mqrhp" (UID: "bb0c72b0-0d55-4136-99ce-d50036c55d91") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:57:14.710576 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:14.710545 2874 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:57:14.710677 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:14.710595 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret podName:cb442e64-9a04-459c-814a-71a5bf49c5f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:30.710580167 +0000 UTC m=+224.044944867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret") pod "global-pull-secret-syncer-t4fzq" (UID: "cb442e64-9a04-459c-814a-71a5bf49c5f7") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:57:14.743276 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:14.743249 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:14.743421 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:14.743257 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:14.743509 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:14.743385 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:14.743509 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:14.743506 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:14.743509 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:14.743491 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:14.743727 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:14.743573 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:15.100584 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.100485 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" event={"ID":"d8254aa7-70d3-4bdc-87a4-457d77cda2e5","Type":"ContainerStarted","Data":"c5d04ffbb3d2e289bf2e485e4746d09a514f94bde4061a9035d284f31850421b"} Sep 18 16:57:15.103165 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.103129 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wd6bk" event={"ID":"3c4d9103-40fb-4c3e-b830-bc31eb5edd90","Type":"ContainerStarted","Data":"22d2cd12d3835bd0d4d97b4afe4f4068da192da771fa332fd17c8fffc8b3d34a"} Sep 18 16:57:15.104943 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.104909 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nn99k" event={"ID":"16cad111-725f-4c32-92e9-114a389ea4c2","Type":"ContainerStarted","Data":"f8a7b6f6c571efade64b7d1a21359ae5696cf97f39c94be94e5ee1eb8157037c"} Sep 18 16:57:15.106418 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.106358 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-br879" event={"ID":"eb9022da-0935-4c13-9fdc-102da03b843e","Type":"ContainerStarted","Data":"19ee56105d111386d0ed8a38303b3c4d86b72a873bfb3d270e56cf3bf3a830bc"} Sep 18 16:57:15.108654 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.108619 2874 generic.go:356] "Generic (PLEG): container finished" podID="e9465c98-ea01-4cd6-83b7-485bfb6b54f0" containerID="1522e1661dd464fc248b5d1827b45c13286a7a125bab5be505fcef10bccf8363" exitCode=0 Sep 18 16:57:15.108881 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.108714 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" event={"ID":"e9465c98-ea01-4cd6-83b7-485bfb6b54f0","Type":"ContainerDied","Data":"1522e1661dd464fc248b5d1827b45c13286a7a125bab5be505fcef10bccf8363"} Sep 18 16:57:15.116955 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.116928 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerStarted","Data":"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac"} Sep 18 16:57:15.117058 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.117009 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerStarted","Data":"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65"} Sep 18 16:57:15.117124 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.117057 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerStarted","Data":"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881"} Sep 18 16:57:15.117124 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.117105 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerStarted","Data":"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f"} Sep 18 16:57:15.117228 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.117150 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerStarted","Data":"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76"} Sep 18 16:57:15.117228 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.117197 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerStarted","Data":"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1"} Sep 18 16:57:15.119809 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.119785 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2hbpw" event={"ID":"16a22de8-89ae-401d-b3ff-8554161ebd3d","Type":"ContainerStarted","Data":"1d2f124272ef65d07ead700c80484fb22eddc58f0b4e26e7dec8b0f8249e1c88"} Sep 18 16:57:15.120537 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.120491 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fn6d2" podStartSLOduration=5.006753972 podStartE2EDuration="20.12047621s" totalImagesPullingTime="15.113722238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:15.120384788 +0000 UTC m=+208.454749497" watchObservedRunningTime="2026-09-18 16:57:15.12047621 +0000 UTC m=+208.454840931" Sep 18 16:57:15.122531 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.122509 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" event={"ID":"b825e74d-e9fc-43d1-91d5-c318fe1dab02","Type":"ContainerStarted","Data":"fb876e453808ce8313d9bf6ded8bc073ce823125069bec6406cdbf5f06253554"} Sep 18 16:57:15.152453 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.152100 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wd6bk" podStartSLOduration=8.932636343 podStartE2EDuration="20.152085381s" totalImagesPullingTime="11.219449038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:15.13609191 +0000 UTC m=+208.470456621" watchObservedRunningTime="2026-09-18 16:57:15.152085381 +0000 UTC m=+208.486450087" Sep 18 16:57:15.185489 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.185441 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2hbpw" podStartSLOduration=4.931584648 podStartE2EDuration="20.18541907s" totalImagesPullingTime="15.253834422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:15.152804948 +0000 UTC m=+208.487169655" watchObservedRunningTime="2026-09-18 16:57:15.18541907 +0000 UTC m=+208.519783775" Sep 18 16:57:15.490532 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.490490 2874 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 18 16:57:15.874575 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.874475 2874 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:57:15.490509234Z","UUID":"3fc61e36-bfce-420e-b954-8ab1cc98085f","Handler":null,"Name":"","Endpoint":""} Sep 18 16:57:15.876031 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.876006 2874 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 18 16:57:15.876173 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:15.876040 2874 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 18 16:57:16.127057 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:16.127024 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" event={"ID":"b825e74d-e9fc-43d1-91d5-c318fe1dab02","Type":"ContainerStarted","Data":"2f08b68e79cc23e9a6639702a1b5c6af4a00d528a2c68eb1549e30ca5c9fa88e"} Sep 18 16:57:16.753976 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:16.751559 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:16.753976 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:16.751728 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:16.753976 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:16.752089 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:16.753976 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:16.752223 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:16.753976 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:16.752277 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:16.753976 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:16.752366 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:16.796209 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:16.796014 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:57:17.130847 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:17.130813 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-br879" event={"ID":"eb9022da-0935-4c13-9fdc-102da03b843e","Type":"ContainerStarted","Data":"acf9cb32c8961fb175b7a1dd8a386bd3ee77511bfe529d17e2716c7a489e3578"} Sep 18 16:57:17.135214 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:17.135178 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerStarted","Data":"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9"} Sep 18 16:57:17.325042 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:17.325013 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dcrn\" (UniqueName: \"kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn\") pod \"network-check-target-npjd7\" (UID: \"e9379c30-e23f-4635-a9c1-73d866893686\") " pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:17.325212 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:17.325166 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:57:17.325212 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:17.325187 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:57:17.325212 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:17.325197 2874 projected.go:196] Error preparing data for projected volume kube-api-access-9dcrn for pod openshift-network-diagnostics/network-check-target-npjd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:17.325346 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:17.325252 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn podName:e9379c30-e23f-4635-a9c1-73d866893686 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:33.325235046 +0000 UTC m=+226.659599745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dcrn" (UniqueName: "kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn") pod "network-check-target-npjd7" (UID: "e9379c30-e23f-4635-a9c1-73d866893686") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:18.140161 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:18.140111 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" event={"ID":"b825e74d-e9fc-43d1-91d5-c318fe1dab02","Type":"ContainerStarted","Data":"b384e245fd24d2cadd7ae7cb1068cc68172b84ffe52eca181dc4b29dbac2cc78"} Sep 18 16:57:18.141888 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:18.141857 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nn99k" event={"ID":"16cad111-725f-4c32-92e9-114a389ea4c2","Type":"ContainerStarted","Data":"f6a83e65be96d88e3ce5e75fdb33e839fa5958bfc046d8981ab7ea7e35257c16"} Sep 18 16:57:18.161218 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:18.161175 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldv6s" podStartSLOduration=5.426822045 podStartE2EDuration="23.161161465s" totalImagesPullingTime="17.73433942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:18.160971294 +0000 UTC m=+211.495336018" watchObservedRunningTime="2026-09-18 16:57:18.161161465 +0000 UTC m=+211.495526169" Sep 18 16:57:18.161664 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:18.161623 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-br879" podStartSLOduration=12.706773669 podStartE2EDuration="15.161617s" totalImagesPullingTime="2.454843331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:17.152634011 +0000 UTC m=+210.486998718" watchObservedRunningTime="2026-09-18 16:57:18.161617 +0000 UTC m=+211.495981707" Sep 18 16:57:18.179825 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:18.179790 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nn99k" podStartSLOduration=11.730995446 podStartE2EDuration="14.179778084s" totalImagesPullingTime="2.448782638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:18.179266891 +0000 UTC m=+211.513631601" watchObservedRunningTime="2026-09-18 16:57:18.179778084 +0000 UTC m=+211.514142790" Sep 18 16:57:18.743812 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:18.743772 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:18.743812 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:18.743805 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:18.744050 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:18.743771 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:18.744050 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:18.743919 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:18.744145 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:18.744048 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:18.744193 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:18.744139 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:19.152480 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:19.152447 2874 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:57:19.153067 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:19.153051 2874 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:57:20.149467 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:20.149215 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" event={"ID":"e9465c98-ea01-4cd6-83b7-485bfb6b54f0","Type":"ContainerStarted","Data":"13a2b11285a124f215f322b1f3cfddbe0b85eec76742b06820bb37cda46baa6e"} Sep 18 16:57:20.153347 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:20.153314 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerStarted","Data":"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2"} Sep 18 16:57:20.154046 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:20.153542 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:57:20.154046 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:20.153987 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wd6bk" Sep 18 16:57:20.194061 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:20.194020 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" podStartSLOduration=7.066540304 podStartE2EDuration="22.194004861s" totalImagesPullingTime="15.127464557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:20.193305388 +0000 UTC m=+213.527670094" watchObservedRunningTime="2026-09-18 16:57:20.194004861 +0000 UTC m=+213.528369567" Sep 18 16:57:20.743838 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:20.743803 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:20.744015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:20.743803 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:20.744015 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:20.743936 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:20.744015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:20.743803 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:20.744015 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:20.743995 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:20.744179 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:20.744072 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:21.156792 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.156760 2874 generic.go:356] "Generic (PLEG): container finished" podID="e9465c98-ea01-4cd6-83b7-485bfb6b54f0" containerID="13a2b11285a124f215f322b1f3cfddbe0b85eec76742b06820bb37cda46baa6e" exitCode=0 Sep 18 16:57:21.157224 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.156848 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" event={"ID":"e9465c98-ea01-4cd6-83b7-485bfb6b54f0","Type":"ContainerDied","Data":"13a2b11285a124f215f322b1f3cfddbe0b85eec76742b06820bb37cda46baa6e"} Sep 18 16:57:21.158015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.157511 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:57:21.158015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.157595 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:57:21.158015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.157648 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:57:21.339596 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.339570 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:57:21.339814 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.339707 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:57:21.797175 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:21.797135 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:57:21.979022 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.978780 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-npjd7"] Sep 18 16:57:21.979190 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.979162 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:21.979298 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:21.979273 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:21.981826 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.981782 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mqrhp"] Sep 18 16:57:21.981952 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.981917 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:21.982057 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:21.982036 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:21.982505 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.982482 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t4fzq"] Sep 18 16:57:21.982603 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:21.982586 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:21.982747 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:21.982697 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:23.163721 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.163695 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_8c271c1096af51a1f9f8e05948cf36ce/setup/9.log" Sep 18 16:57:23.164336 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.163732 2874 generic.go:356] "Generic (PLEG): container finished" podID="8c271c1096af51a1f9f8e05948cf36ce" containerID="2366444a625976b42c6701197922a0af4e8a5ac599f00959579eab674ba00751" exitCode=0 Sep 18 16:57:23.164336 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.163791 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerDied","Data":"2366444a625976b42c6701197922a0af4e8a5ac599f00959579eab674ba00751"} Sep 18 16:57:23.164336 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.163838 2874 scope.go:117] "RemoveContainer" containerID="651c540d09203e2f13cf25ddc0be8e5e06bfc1ec0568fba943b0d72ea4c5b093" Sep 18 16:57:23.164336 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.164114 2874 scope.go:117] "RemoveContainer" containerID="651c540d09203e2f13cf25ddc0be8e5e06bfc1ec0568fba943b0d72ea4c5b093" Sep 18 16:57:23.165990 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.165966 2874 generic.go:356] "Generic (PLEG): container finished" podID="e9465c98-ea01-4cd6-83b7-485bfb6b54f0" containerID="473145cab29ff9edcce29b0b3c69c066113aaa9b01b5e741b282363b7291192e" exitCode=0 Sep 18 16:57:23.166075 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.166054 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" event={"ID":"e9465c98-ea01-4cd6-83b7-485bfb6b54f0","Type":"ContainerDied","Data":"473145cab29ff9edcce29b0b3c69c066113aaa9b01b5e741b282363b7291192e"} Sep 18 16:57:23.193960 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:23.193764 2874 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_9 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: '651c540d09203e2f13cf25ddc0be8e5e06bfc1ec0568fba943b0d72ea4c5b093'" containerID="651c540d09203e2f13cf25ddc0be8e5e06bfc1ec0568fba943b0d72ea4c5b093" Sep 18 16:57:23.193960 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:23.193805 2874 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator_8c271c1096af51a1f9f8e05948cf36ce_9 in pod sandbox 4dd722bf63f7bf4bb0ccb7b957c8c09cf2bea4644630bcaa94fe3e1af6f3b875 from index: no such id: '651c540d09203e2f13cf25ddc0be8e5e06bfc1ec0568fba943b0d72ea4c5b093'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal_openshift-machine-config-operator(8c271c1096af51a1f9f8e05948cf36ce)\"" logger="UnhandledError" Sep 18 16:57:23.421752 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.421721 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-4xfr7"] Sep 18 16:57:23.743348 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.743313 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:23.743531 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.743313 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:23.743531 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:23.743464 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:23.743531 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:23.743480 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:23.743531 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:23.743330 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:23.743703 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:23.743571 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:24.170059 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.170021 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" event={"ID":"8c271c1096af51a1f9f8e05948cf36ce","Type":"ContainerStarted","Data":"2983a44c8de3bd12e25dd92651c50728b454cca229e0a5cc87113e1f054d9d6d"} Sep 18 16:57:24.172692 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.172653 2874 generic.go:356] "Generic (PLEG): container finished" podID="e9465c98-ea01-4cd6-83b7-485bfb6b54f0" containerID="277960618d4ad4c26389fa66b92f2257500d801db7a24402405a2cbb957928f9" exitCode=0 Sep 18 16:57:24.172800 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.172717 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" event={"ID":"e9465c98-ea01-4cd6-83b7-485bfb6b54f0","Type":"ContainerDied","Data":"277960618d4ad4c26389fa66b92f2257500d801db7a24402405a2cbb957928f9"} Sep 18 16:57:24.172928 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.172904 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="nbdb" containerID="cri-o://716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac" gracePeriod=30 Sep 18 16:57:24.173157 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.172929 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="kube-rbac-proxy-ovn-metrics" containerID="cri-o://17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881" gracePeriod=30 Sep 18 16:57:24.173157 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.172959 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovn-acl-logging" containerID="cri-o://c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76" gracePeriod=30 Sep 18 16:57:24.173157 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.172969 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="northd" containerID="cri-o://a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65" gracePeriod=30 Sep 18 16:57:24.173157 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.172907 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovn-controller" containerID="cri-o://775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1" gracePeriod=30 Sep 18 16:57:24.173157 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.173028 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="kube-rbac-proxy-node" containerID="cri-o://ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f" gracePeriod=30 Sep 18 16:57:24.173157 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.172970 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="sbdb" containerID="cri-o://00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9" gracePeriod=30 Sep 18 16:57:24.185987 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.185795 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovnkube-controller" containerID="cri-o://9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2" gracePeriod=30 Sep 18 16:57:24.191839 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.191794 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-99.ec2.internal" podStartSLOduration=26.191777051 podStartE2EDuration="26.191777051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:24.191300496 +0000 UTC m=+217.525665200" watchObservedRunningTime="2026-09-18 16:57:24.191777051 +0000 UTC m=+217.526141761" Sep 18 16:57:24.197216 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:24.197093 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovnkube-controller" probeResult="failure" output="" Sep 18 16:57:25.038995 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.038960 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4xfr7_3a5603d8-009a-4758-ad7b-421e5da3a9f3/ovnkube-controller/0.log" Sep 18 16:57:25.040579 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.040548 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4xfr7_3a5603d8-009a-4758-ad7b-421e5da3a9f3/kube-rbac-proxy-ovn-metrics/0.log" Sep 18 16:57:25.041096 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.041015 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4xfr7_3a5603d8-009a-4758-ad7b-421e5da3a9f3/kube-rbac-proxy-node/0.log" Sep 18 16:57:25.041473 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.041457 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4xfr7_3a5603d8-009a-4758-ad7b-421e5da3a9f3/ovn-acl-logging/0.log" Sep 18 16:57:25.042013 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.041996 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4xfr7_3a5603d8-009a-4758-ad7b-421e5da3a9f3/ovn-controller/0.log" Sep 18 16:57:25.063933 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.063905 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:57:25.074968 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.074946 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-systemd-units\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.074968 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.074973 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-netd\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.074986 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-netns\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075008 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-openvswitch\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075028 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-systemd\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075052 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-netd" (OuterVolumeSpecName: "host-cni-netd") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "host-cni-netd". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075062 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-config\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075053 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-systemd-units" (OuterVolumeSpecName: "systemd-units") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "systemd-units". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075073 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-netns" (OuterVolumeSpecName: "host-run-netns") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "host-run-netns". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075086 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovn-node-metrics-cert\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075082 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-openvswitch" (OuterVolumeSpecName: "run-openvswitch") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "run-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075128 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-script-lib\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075152 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-var-lib-openvswitch\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075180 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-env-overrides\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075203 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-ovn-kubernetes\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075213 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-var-lib-openvswitch" (OuterVolumeSpecName: "var-lib-openvswitch") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "var-lib-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075235 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-kubelet\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075246 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-ovn-kubernetes" (OuterVolumeSpecName: "host-run-ovn-kubernetes") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "host-run-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075260 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-slash\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075282 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-log-socket\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075310 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-node-log\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075281 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-kubelet" (OuterVolumeSpecName: "host-kubelet") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "host-kubelet". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075332 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-log-socket" (OuterVolumeSpecName: "log-socket") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "log-socket". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075305 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-slash" (OuterVolumeSpecName: "host-slash") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "host-slash". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075337 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlc7m\" (UniqueName: \"kubernetes.io/projected/3a5603d8-009a-4758-ad7b-421e5da3a9f3-kube-api-access-dlc7m\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075374 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-node-log" (OuterVolumeSpecName: "node-log") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "node-log". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075383 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-bin\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075423 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-bin" (OuterVolumeSpecName: "host-cni-bin") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "host-cni-bin". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.075568 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075451 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075480 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-etc-openvswitch\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075503 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-ovn\") pod \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\" (UID: \"3a5603d8-009a-4758-ad7b-421e5da3a9f3\") " Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075521 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-var-lib-cni-networks-ovn-kubernetes" (OuterVolumeSpecName: "host-var-lib-cni-networks-ovn-kubernetes") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "host-var-lib-cni-networks-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075539 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-env-overrides" (OuterVolumeSpecName: "env-overrides") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "env-overrides". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075547 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-etc-openvswitch" (OuterVolumeSpecName: "etc-openvswitch") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "etc-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075547 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-config" (OuterVolumeSpecName: "ovnkube-config") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "ovnkube-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075571 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-ovn" (OuterVolumeSpecName: "run-ovn") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "run-ovn". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075590 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-script-lib" (OuterVolumeSpecName: "ovnkube-script-lib") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "ovnkube-script-lib". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075662 2874 reconciler_common.go:299] "Volume detached for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-kubelet\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075674 2874 reconciler_common.go:299] "Volume detached for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-slash\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075682 2874 reconciler_common.go:299] "Volume detached for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-log-socket\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075689 2874 reconciler_common.go:299] "Volume detached for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-node-log\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075697 2874 reconciler_common.go:299] "Volume detached for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-bin\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075704 2874 reconciler_common.go:299] "Volume detached for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-var-lib-cni-networks-ovn-kubernetes\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075712 2874 reconciler_common.go:299] "Volume detached for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-etc-openvswitch\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075720 2874 reconciler_common.go:299] "Volume detached for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-ovn\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075728 2874 reconciler_common.go:299] "Volume detached for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-systemd-units\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075737 2874 reconciler_common.go:299] "Volume detached for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-cni-netd\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075744 2874 reconciler_common.go:299] "Volume detached for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-netns\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075752 2874 reconciler_common.go:299] "Volume detached for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-openvswitch\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076861 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075759 2874 reconciler_common.go:299] "Volume detached for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076861 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075766 2874 reconciler_common.go:299] "Volume detached for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovnkube-script-lib\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076861 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075773 2874 reconciler_common.go:299] "Volume detached for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-var-lib-openvswitch\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076861 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075781 2874 reconciler_common.go:299] "Volume detached for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a5603d8-009a-4758-ad7b-421e5da3a9f3-env-overrides\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.076861 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.075789 2874 reconciler_common.go:299] "Volume detached for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-host-run-ovn-kubernetes\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.078817 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.078791 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovn-node-metrics-cert" (OuterVolumeSpecName: "ovn-node-metrics-cert") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "ovn-node-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:57:25.078970 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.078951 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5603d8-009a-4758-ad7b-421e5da3a9f3-kube-api-access-dlc7m" (OuterVolumeSpecName: "kube-api-access-dlc7m") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "kube-api-access-dlc7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:57:25.079646 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.079627 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-systemd" (OuterVolumeSpecName: "run-systemd") pod "3a5603d8-009a-4758-ad7b-421e5da3a9f3" (UID: "3a5603d8-009a-4758-ad7b-421e5da3a9f3"). InnerVolumeSpecName "run-systemd". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:57:25.119089 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119033 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-gsvlw"] Sep 18 16:57:25.119198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119159 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovnkube-controller" Sep 18 16:57:25.119198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119170 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovnkube-controller" Sep 18 16:57:25.119198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119180 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovn-controller" Sep 18 16:57:25.119198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119186 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovn-controller" Sep 18 16:57:25.119198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119193 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="kube-rbac-proxy-node" Sep 18 16:57:25.119198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119199 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="kube-rbac-proxy-node" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119204 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="northd" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119210 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="northd" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119217 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="sbdb" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119222 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="sbdb" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119228 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovn-acl-logging" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119233 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovn-acl-logging" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119239 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="kube-rbac-proxy-ovn-metrics" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119245 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="kube-rbac-proxy-ovn-metrics" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119249 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="nbdb" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119254 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="nbdb" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119279 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovnkube-controller" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119285 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="kube-rbac-proxy-ovn-metrics" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119290 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="northd" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119295 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="nbdb" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119300 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="sbdb" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119304 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovn-controller" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119309 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="ovn-acl-logging" Sep 18 16:57:25.119371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.119313 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerName="kube-rbac-proxy-node" Sep 18 16:57:25.136861 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.136831 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176128 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176096 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-etc-openvswitch\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176128 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176134 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-slash\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176152 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-run-netns\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176174 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-cni-bin\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176194 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-run-systemd\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176216 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/74a7c0f8-bcb1-452f-8989-da958fe9ce25-env-overrides\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176265 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176283 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/74a7c0f8-bcb1-452f-8989-da958fe9ce25-ovnkube-script-lib\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176303 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-systemd-units\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176326 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-run-openvswitch\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176373 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-run-ovn\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176414 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/74a7c0f8-bcb1-452f-8989-da958fe9ce25-ovn-node-metrics-cert\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176454 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-kubelet\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176482 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176508 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/74a7c0f8-bcb1-452f-8989-da958fe9ce25-ovnkube-config\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176534 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-var-lib-openvswitch\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176555 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-log-socket\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.176604 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176581 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-node-log\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.177226 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176604 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7wj4\" (UniqueName: \"kubernetes.io/projected/74a7c0f8-bcb1-452f-8989-da958fe9ce25-kube-api-access-q7wj4\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.177226 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176632 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-cni-netd\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.177226 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176718 2874 reconciler_common.go:299] "Volume detached for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a5603d8-009a-4758-ad7b-421e5da3a9f3-run-systemd\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.177226 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176743 2874 reconciler_common.go:299] "Volume detached for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a5603d8-009a-4758-ad7b-421e5da3a9f3-ovn-node-metrics-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.177226 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.176843 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlc7m\" (UniqueName: \"kubernetes.io/projected/3a5603d8-009a-4758-ad7b-421e5da3a9f3-kube-api-access-dlc7m\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:57:25.177438 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.177304 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4xfr7_3a5603d8-009a-4758-ad7b-421e5da3a9f3/ovnkube-controller/0.log" Sep 18 16:57:25.179197 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.179177 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4xfr7_3a5603d8-009a-4758-ad7b-421e5da3a9f3/kube-rbac-proxy-ovn-metrics/0.log" Sep 18 16:57:25.179664 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.179638 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4xfr7_3a5603d8-009a-4758-ad7b-421e5da3a9f3/kube-rbac-proxy-node/0.log" Sep 18 16:57:25.180209 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180188 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4xfr7_3a5603d8-009a-4758-ad7b-421e5da3a9f3/ovn-acl-logging/0.log" Sep 18 16:57:25.180692 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180678 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4xfr7_3a5603d8-009a-4758-ad7b-421e5da3a9f3/ovn-controller/0.log" Sep 18 16:57:25.180780 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180720 2874 generic.go:356] "Generic (PLEG): container finished" podID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerID="9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2" exitCode=1 Sep 18 16:57:25.180780 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180737 2874 generic.go:356] "Generic (PLEG): container finished" podID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerID="00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9" exitCode=0 Sep 18 16:57:25.180780 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180742 2874 generic.go:356] "Generic (PLEG): container finished" podID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerID="716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac" exitCode=0 Sep 18 16:57:25.180780 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180748 2874 generic.go:356] "Generic (PLEG): container finished" podID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerID="a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65" exitCode=0 Sep 18 16:57:25.180780 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180753 2874 generic.go:356] "Generic (PLEG): container finished" podID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerID="17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881" exitCode=143 Sep 18 16:57:25.180780 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180758 2874 generic.go:356] "Generic (PLEG): container finished" podID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerID="ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f" exitCode=143 Sep 18 16:57:25.180780 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180763 2874 generic.go:356] "Generic (PLEG): container finished" podID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerID="c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76" exitCode=143 Sep 18 16:57:25.180780 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180767 2874 generic.go:356] "Generic (PLEG): container finished" podID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" containerID="775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1" exitCode=143 Sep 18 16:57:25.181168 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180812 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerDied","Data":"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2"} Sep 18 16:57:25.181168 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180832 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" Sep 18 16:57:25.181168 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180880 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerDied","Data":"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9"} Sep 18 16:57:25.181168 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180928 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerDied","Data":"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac"} Sep 18 16:57:25.181168 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180957 2874 scope.go:117] "RemoveContainer" containerID="9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2" Sep 18 16:57:25.181168 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.180974 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerDied","Data":"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65"} Sep 18 16:57:25.181168 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181021 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerDied","Data":"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881"} Sep 18 16:57:25.181168 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181069 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerDied","Data":"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f"} Sep 18 16:57:25.181168 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181115 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerDied","Data":"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181159 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181181 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181186 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181193 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181199 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181210 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerDied","Data":"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181241 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181247 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181251 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181256 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181260 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181263 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181268 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181274 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181283 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xfr7" event={"ID":"3a5603d8-009a-4758-ad7b-421e5da3a9f3","Type":"ContainerDied","Data":"aee3d358ab3c285eec71e2cfab0cc6ebb979e815da31cac34d5d216f748c7b50"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181338 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181347 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181354 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181360 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181366 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181372 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181379 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76"} Sep 18 16:57:25.181625 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.181385 2874 pod_container_deletor.go:114] "Failed to issue the request to remove container" containerID={"Type":"cri-o","ID":"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1"} Sep 18 16:57:25.192888 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.192872 2874 scope.go:117] "RemoveContainer" containerID="00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9" Sep 18 16:57:25.200611 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.200594 2874 scope.go:117] "RemoveContainer" containerID="716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac" Sep 18 16:57:25.207647 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.207632 2874 scope.go:117] "RemoveContainer" containerID="a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65" Sep 18 16:57:25.213925 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.213905 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-4xfr7"] Sep 18 16:57:25.225269 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.225243 2874 scope.go:117] "RemoveContainer" containerID="17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881" Sep 18 16:57:25.227339 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.225367 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-4xfr7"] Sep 18 16:57:25.232910 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.232891 2874 scope.go:117] "RemoveContainer" containerID="ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f" Sep 18 16:57:25.241816 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.241800 2874 scope.go:117] "RemoveContainer" containerID="c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76" Sep 18 16:57:25.247371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.247355 2874 scope.go:117] "RemoveContainer" containerID="775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1" Sep 18 16:57:25.253431 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.253416 2874 scope.go:117] "RemoveContainer" containerID="9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2" Sep 18 16:57:25.253699 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.253683 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": container with ID starting with 9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2 not found: ID does not exist" containerID="9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2" Sep 18 16:57:25.253736 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.253707 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2"} err="failed to get container status \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": rpc error: code = NotFound desc = could not find container \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": container with ID starting with 9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2 not found: ID does not exist" Sep 18 16:57:25.253736 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.253723 2874 scope.go:117] "RemoveContainer" containerID="00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9" Sep 18 16:57:25.253954 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.253941 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": container with ID starting with 00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9 not found: ID does not exist" containerID="00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9" Sep 18 16:57:25.253987 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.253958 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9"} err="failed to get container status \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": rpc error: code = NotFound desc = could not find container \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": container with ID starting with 00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9 not found: ID does not exist" Sep 18 16:57:25.253987 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.253970 2874 scope.go:117] "RemoveContainer" containerID="716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac" Sep 18 16:57:25.254154 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.254138 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": container with ID starting with 716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac not found: ID does not exist" containerID="716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac" Sep 18 16:57:25.254199 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.254158 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac"} err="failed to get container status \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": rpc error: code = NotFound desc = could not find container \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": container with ID starting with 716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac not found: ID does not exist" Sep 18 16:57:25.254199 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.254169 2874 scope.go:117] "RemoveContainer" containerID="a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65" Sep 18 16:57:25.254353 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.254338 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": container with ID starting with a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65 not found: ID does not exist" containerID="a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65" Sep 18 16:57:25.254388 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.254356 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65"} err="failed to get container status \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": rpc error: code = NotFound desc = could not find container \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": container with ID starting with a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65 not found: ID does not exist" Sep 18 16:57:25.254388 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.254366 2874 scope.go:117] "RemoveContainer" containerID="17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881" Sep 18 16:57:25.254627 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.254610 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": container with ID starting with 17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881 not found: ID does not exist" containerID="17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881" Sep 18 16:57:25.254668 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.254631 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881"} err="failed to get container status \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": rpc error: code = NotFound desc = could not find container \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": container with ID starting with 17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881 not found: ID does not exist" Sep 18 16:57:25.254668 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.254644 2874 scope.go:117] "RemoveContainer" containerID="ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f" Sep 18 16:57:25.254860 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.254846 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": container with ID starting with ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f not found: ID does not exist" containerID="ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f" Sep 18 16:57:25.254899 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.254863 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f"} err="failed to get container status \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": rpc error: code = NotFound desc = could not find container \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": container with ID starting with ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f not found: ID does not exist" Sep 18 16:57:25.254899 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.254873 2874 scope.go:117] "RemoveContainer" containerID="c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76" Sep 18 16:57:25.255105 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.255090 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": container with ID starting with c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76 not found: ID does not exist" containerID="c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76" Sep 18 16:57:25.255152 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.255105 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76"} err="failed to get container status \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": rpc error: code = NotFound desc = could not find container \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": container with ID starting with c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76 not found: ID does not exist" Sep 18 16:57:25.255152 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.255115 2874 scope.go:117] "RemoveContainer" containerID="775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1" Sep 18 16:57:25.255307 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.255294 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": container with ID starting with 775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1 not found: ID does not exist" containerID="775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1" Sep 18 16:57:25.255348 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.255309 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1"} err="failed to get container status \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": rpc error: code = NotFound desc = could not find container \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": container with ID starting with 775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1 not found: ID does not exist" Sep 18 16:57:25.255348 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.255320 2874 scope.go:117] "RemoveContainer" containerID="9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2" Sep 18 16:57:25.255557 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.255541 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2"} err="failed to get container status \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": rpc error: code = NotFound desc = could not find container \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": container with ID starting with 9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2 not found: ID does not exist" Sep 18 16:57:25.255557 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.255556 2874 scope.go:117] "RemoveContainer" containerID="00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9" Sep 18 16:57:25.255782 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.255760 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9"} err="failed to get container status \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": rpc error: code = NotFound desc = could not find container \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": container with ID starting with 00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9 not found: ID does not exist" Sep 18 16:57:25.255826 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.255783 2874 scope.go:117] "RemoveContainer" containerID="716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac" Sep 18 16:57:25.255998 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.255984 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac"} err="failed to get container status \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": rpc error: code = NotFound desc = could not find container \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": container with ID starting with 716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac not found: ID does not exist" Sep 18 16:57:25.255998 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.255997 2874 scope.go:117] "RemoveContainer" containerID="a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65" Sep 18 16:57:25.256194 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.256174 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65"} err="failed to get container status \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": rpc error: code = NotFound desc = could not find container \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": container with ID starting with a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65 not found: ID does not exist" Sep 18 16:57:25.256239 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.256195 2874 scope.go:117] "RemoveContainer" containerID="17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881" Sep 18 16:57:25.256463 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.256447 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881"} err="failed to get container status \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": rpc error: code = NotFound desc = could not find container \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": container with ID starting with 17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881 not found: ID does not exist" Sep 18 16:57:25.256550 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.256464 2874 scope.go:117] "RemoveContainer" containerID="ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f" Sep 18 16:57:25.256714 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.256698 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f"} err="failed to get container status \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": rpc error: code = NotFound desc = could not find container \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": container with ID starting with ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f not found: ID does not exist" Sep 18 16:57:25.256750 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.256716 2874 scope.go:117] "RemoveContainer" containerID="c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76" Sep 18 16:57:25.256937 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.256922 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76"} err="failed to get container status \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": rpc error: code = NotFound desc = could not find container \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": container with ID starting with c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76 not found: ID does not exist" Sep 18 16:57:25.256976 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.256939 2874 scope.go:117] "RemoveContainer" containerID="775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1" Sep 18 16:57:25.257141 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.257126 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1"} err="failed to get container status \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": rpc error: code = NotFound desc = could not find container \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": container with ID starting with 775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1 not found: ID does not exist" Sep 18 16:57:25.257180 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.257142 2874 scope.go:117] "RemoveContainer" containerID="9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2" Sep 18 16:57:25.257351 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.257335 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2"} err="failed to get container status \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": rpc error: code = NotFound desc = could not find container \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": container with ID starting with 9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2 not found: ID does not exist" Sep 18 16:57:25.257386 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.257352 2874 scope.go:117] "RemoveContainer" containerID="00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9" Sep 18 16:57:25.257611 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.257596 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9"} err="failed to get container status \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": rpc error: code = NotFound desc = could not find container \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": container with ID starting with 00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9 not found: ID does not exist" Sep 18 16:57:25.257651 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.257611 2874 scope.go:117] "RemoveContainer" containerID="716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac" Sep 18 16:57:25.257838 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.257821 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac"} err="failed to get container status \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": rpc error: code = NotFound desc = could not find container \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": container with ID starting with 716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac not found: ID does not exist" Sep 18 16:57:25.257884 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.257839 2874 scope.go:117] "RemoveContainer" containerID="a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65" Sep 18 16:57:25.258045 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.258030 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65"} err="failed to get container status \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": rpc error: code = NotFound desc = could not find container \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": container with ID starting with a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65 not found: ID does not exist" Sep 18 16:57:25.258083 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.258045 2874 scope.go:117] "RemoveContainer" containerID="17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881" Sep 18 16:57:25.258262 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.258245 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881"} err="failed to get container status \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": rpc error: code = NotFound desc = could not find container \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": container with ID starting with 17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881 not found: ID does not exist" Sep 18 16:57:25.258303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.258263 2874 scope.go:117] "RemoveContainer" containerID="ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f" Sep 18 16:57:25.258521 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.258503 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f"} err="failed to get container status \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": rpc error: code = NotFound desc = could not find container \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": container with ID starting with ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f not found: ID does not exist" Sep 18 16:57:25.258579 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.258521 2874 scope.go:117] "RemoveContainer" containerID="c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76" Sep 18 16:57:25.258734 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.258712 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76"} err="failed to get container status \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": rpc error: code = NotFound desc = could not find container \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": container with ID starting with c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76 not found: ID does not exist" Sep 18 16:57:25.258734 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.258728 2874 scope.go:117] "RemoveContainer" containerID="775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1" Sep 18 16:57:25.258949 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.258933 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1"} err="failed to get container status \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": rpc error: code = NotFound desc = could not find container \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": container with ID starting with 775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1 not found: ID does not exist" Sep 18 16:57:25.259009 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.258950 2874 scope.go:117] "RemoveContainer" containerID="9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2" Sep 18 16:57:25.259153 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.259138 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2"} err="failed to get container status \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": rpc error: code = NotFound desc = could not find container \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": container with ID starting with 9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2 not found: ID does not exist" Sep 18 16:57:25.259191 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.259152 2874 scope.go:117] "RemoveContainer" containerID="00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9" Sep 18 16:57:25.259355 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.259334 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9"} err="failed to get container status \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": rpc error: code = NotFound desc = could not find container \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": container with ID starting with 00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9 not found: ID does not exist" Sep 18 16:57:25.259396 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.259356 2874 scope.go:117] "RemoveContainer" containerID="716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac" Sep 18 16:57:25.259579 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.259558 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac"} err="failed to get container status \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": rpc error: code = NotFound desc = could not find container \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": container with ID starting with 716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac not found: ID does not exist" Sep 18 16:57:25.259637 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.259580 2874 scope.go:117] "RemoveContainer" containerID="a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65" Sep 18 16:57:25.259800 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.259781 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65"} err="failed to get container status \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": rpc error: code = NotFound desc = could not find container \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": container with ID starting with a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65 not found: ID does not exist" Sep 18 16:57:25.259839 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.259802 2874 scope.go:117] "RemoveContainer" containerID="17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881" Sep 18 16:57:25.260009 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.259993 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881"} err="failed to get container status \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": rpc error: code = NotFound desc = could not find container \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": container with ID starting with 17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881 not found: ID does not exist" Sep 18 16:57:25.260046 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.260011 2874 scope.go:117] "RemoveContainer" containerID="ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f" Sep 18 16:57:25.260267 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.260237 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f"} err="failed to get container status \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": rpc error: code = NotFound desc = could not find container \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": container with ID starting with ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f not found: ID does not exist" Sep 18 16:57:25.260316 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.260269 2874 scope.go:117] "RemoveContainer" containerID="c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76" Sep 18 16:57:25.260575 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.260559 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76"} err="failed to get container status \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": rpc error: code = NotFound desc = could not find container \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": container with ID starting with c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76 not found: ID does not exist" Sep 18 16:57:25.260624 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.260575 2874 scope.go:117] "RemoveContainer" containerID="775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1" Sep 18 16:57:25.260819 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.260801 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1"} err="failed to get container status \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": rpc error: code = NotFound desc = could not find container \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": container with ID starting with 775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1 not found: ID does not exist" Sep 18 16:57:25.260893 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.260821 2874 scope.go:117] "RemoveContainer" containerID="9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2" Sep 18 16:57:25.261080 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.261060 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2"} err="failed to get container status \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": rpc error: code = NotFound desc = could not find container \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": container with ID starting with 9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2 not found: ID does not exist" Sep 18 16:57:25.261123 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.261082 2874 scope.go:117] "RemoveContainer" containerID="00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9" Sep 18 16:57:25.261303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.261278 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9"} err="failed to get container status \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": rpc error: code = NotFound desc = could not find container \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": container with ID starting with 00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9 not found: ID does not exist" Sep 18 16:57:25.261303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.261296 2874 scope.go:117] "RemoveContainer" containerID="716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac" Sep 18 16:57:25.261537 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.261521 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac"} err="failed to get container status \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": rpc error: code = NotFound desc = could not find container \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": container with ID starting with 716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac not found: ID does not exist" Sep 18 16:57:25.261596 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.261538 2874 scope.go:117] "RemoveContainer" containerID="a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65" Sep 18 16:57:25.261761 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.261739 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65"} err="failed to get container status \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": rpc error: code = NotFound desc = could not find container \"a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65\": container with ID starting with a7bca9cb0b08b5042d6aad13dc1797c788cdef8d71870fd47ccb92236b463c65 not found: ID does not exist" Sep 18 16:57:25.261799 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.261763 2874 scope.go:117] "RemoveContainer" containerID="17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881" Sep 18 16:57:25.261964 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.261948 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881"} err="failed to get container status \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": rpc error: code = NotFound desc = could not find container \"17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881\": container with ID starting with 17041ece7173b23c8cacadb00b0ca05ceb17a47b3a7234f6c0b06a03969af881 not found: ID does not exist" Sep 18 16:57:25.262003 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.261964 2874 scope.go:117] "RemoveContainer" containerID="ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f" Sep 18 16:57:25.262154 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.262133 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f"} err="failed to get container status \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": rpc error: code = NotFound desc = could not find container \"ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f\": container with ID starting with ced82379af4da6fc98d392817e092e5268fa81601136fb8cdcff7fe56fa7cf8f not found: ID does not exist" Sep 18 16:57:25.262224 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.262155 2874 scope.go:117] "RemoveContainer" containerID="c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76" Sep 18 16:57:25.262393 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.262375 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76"} err="failed to get container status \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": rpc error: code = NotFound desc = could not find container \"c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76\": container with ID starting with c8a4e523c591723c561ed8d43dc083a32082add5603a9b20af6035fd7ebdfe76 not found: ID does not exist" Sep 18 16:57:25.262466 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.262395 2874 scope.go:117] "RemoveContainer" containerID="775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1" Sep 18 16:57:25.262672 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.262647 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1"} err="failed to get container status \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": rpc error: code = NotFound desc = could not find container \"775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1\": container with ID starting with 775a3f4aa208c414b64202f487a28a9379b0614eddcff37d49395632349474d1 not found: ID does not exist" Sep 18 16:57:25.262672 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.262672 2874 scope.go:117] "RemoveContainer" containerID="9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2" Sep 18 16:57:25.262886 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.262870 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2"} err="failed to get container status \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": rpc error: code = NotFound desc = could not find container \"9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2\": container with ID starting with 9e9a680a26c8e22bcede68ba00406edef2b89a40ec72b82e95cc5fcb00c159b2 not found: ID does not exist" Sep 18 16:57:25.262938 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.262887 2874 scope.go:117] "RemoveContainer" containerID="00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9" Sep 18 16:57:25.263077 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.263062 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9"} err="failed to get container status \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": rpc error: code = NotFound desc = could not find container \"00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9\": container with ID starting with 00c1a41cdba8fe205beb70b0bacdb59e980ba16db2a068eba7bc049be962f7d9 not found: ID does not exist" Sep 18 16:57:25.263118 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.263077 2874 scope.go:117] "RemoveContainer" containerID="716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac" Sep 18 16:57:25.263301 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.263283 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac"} err="failed to get container status \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": rpc error: code = NotFound desc = could not find container \"716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac\": container with ID starting with 716454a0d8dfe2f07e3bd8cc5ec0a884cf80639e3685c60e1183599ec1d0dbac not found: ID does not exist" Sep 18 16:57:25.277668 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277645 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-cni-netd\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.277722 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277687 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-etc-openvswitch\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.277722 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277705 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-slash\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.277803 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277724 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-run-netns\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.277803 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277749 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-cni-bin\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.277803 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277759 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-cni-netd\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.277803 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277759 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-etc-openvswitch\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.277803 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277781 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-slash\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.277803 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277789 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-run-netns\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277771 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-run-systemd\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277798 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-cni-bin\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277816 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-run-systemd\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277823 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/74a7c0f8-bcb1-452f-8989-da958fe9ce25-env-overrides\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277859 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277875 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/74a7c0f8-bcb1-452f-8989-da958fe9ce25-ovnkube-script-lib\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277888 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-systemd-units\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277903 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-run-openvswitch\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277933 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-run-ovn\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277948 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277956 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/74a7c0f8-bcb1-452f-8989-da958fe9ce25-ovn-node-metrics-cert\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277967 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-run-openvswitch\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278003 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-run-ovn\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278014 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-kubelet\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.277960 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-systemd-units\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278041 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-kubelet\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278044 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278078 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/74a7c0f8-bcb1-452f-8989-da958fe9ce25-ovnkube-config\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278079 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278098 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-var-lib-openvswitch\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278124 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-log-socket\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278140 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-node-log\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278156 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7wj4\" (UniqueName: \"kubernetes.io/projected/74a7c0f8-bcb1-452f-8989-da958fe9ce25-kube-api-access-q7wj4\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278157 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-var-lib-openvswitch\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278217 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-log-socket\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278250 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/74a7c0f8-bcb1-452f-8989-da958fe9ce25-node-log\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278293 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/74a7c0f8-bcb1-452f-8989-da958fe9ce25-env-overrides\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278430 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/74a7c0f8-bcb1-452f-8989-da958fe9ce25-ovnkube-script-lib\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.278735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.278596 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/74a7c0f8-bcb1-452f-8989-da958fe9ce25-ovnkube-config\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.280237 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.280220 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/74a7c0f8-bcb1-452f-8989-da958fe9ce25-ovn-node-metrics-cert\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.295493 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.295466 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7wj4\" (UniqueName: \"kubernetes.io/projected/74a7c0f8-bcb1-452f-8989-da958fe9ce25-kube-api-access-q7wj4\") pod \"ovnkube-node-gsvlw\" (UID: \"74a7c0f8-bcb1-452f-8989-da958fe9ce25\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.446599 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.446566 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:25.489450 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:25.489420 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a7c0f8_bcb1_452f_8989_da958fe9ce25.slice/crio-2a4ddfdfb6c504637b7cb49175948187813a4a9b55e60d978c32a275d7b79c1a WatchSource:0}: Error finding container 2a4ddfdfb6c504637b7cb49175948187813a4a9b55e60d978c32a275d7b79c1a: Status 404 returned error can't find the container with id 2a4ddfdfb6c504637b7cb49175948187813a4a9b55e60d978c32a275d7b79c1a Sep 18 16:57:25.743550 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.743507 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:25.743550 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.743530 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:25.743725 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:25.743508 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:25.743725 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.743639 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:25.743795 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.743743 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:25.743840 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:25.743820 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:26.187911 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:26.187579 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" event={"ID":"74a7c0f8-bcb1-452f-8989-da958fe9ce25","Type":"ContainerStarted","Data":"199f2717869a11518f10aaf5256a1065e24e8fd0e02862c4397652e84abe795a"} Sep 18 16:57:26.187911 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:26.187661 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" event={"ID":"74a7c0f8-bcb1-452f-8989-da958fe9ce25","Type":"ContainerStarted","Data":"80e1801bd84ef6c1901edb8a2b922e1fc76ceac8622b463afd31291ad07996a7"} Sep 18 16:57:26.187911 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:26.187693 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" event={"ID":"74a7c0f8-bcb1-452f-8989-da958fe9ce25","Type":"ContainerStarted","Data":"83043eced432a5bfa9cdef58f7355c0abf8d91d89243aedd14d60a6c438d221c"} Sep 18 16:57:26.187911 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:26.187722 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" event={"ID":"74a7c0f8-bcb1-452f-8989-da958fe9ce25","Type":"ContainerStarted","Data":"3cd8d42f0ad74dcb43a80260dfea4c0015b1e15733c64c65926f62a6f7d0f278"} Sep 18 16:57:26.187911 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:26.187754 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" event={"ID":"74a7c0f8-bcb1-452f-8989-da958fe9ce25","Type":"ContainerStarted","Data":"b99140a2111cccba933c857732a8345a0bbb705264f651febfbc25b9efc686d9"} Sep 18 16:57:26.187911 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:26.187803 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" event={"ID":"74a7c0f8-bcb1-452f-8989-da958fe9ce25","Type":"ContainerStarted","Data":"ac08706b548e290a250f4460da894f9734cb12727ee2a04d819005035299ff11"} Sep 18 16:57:26.187911 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:26.187848 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" event={"ID":"74a7c0f8-bcb1-452f-8989-da958fe9ce25","Type":"ContainerStarted","Data":"2a4ddfdfb6c504637b7cb49175948187813a4a9b55e60d978c32a275d7b79c1a"} Sep 18 16:57:26.746987 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:26.746947 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a5603d8-009a-4758-ad7b-421e5da3a9f3" path="/var/lib/kubelet/pods/3a5603d8-009a-4758-ad7b-421e5da3a9f3/volumes" Sep 18 16:57:26.797881 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:26.797842 2874 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" Sep 18 16:57:27.743088 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:27.742886 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:27.743581 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:27.742916 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:27.743581 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:27.743185 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:27.743581 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:27.743297 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:27.743581 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:27.742960 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:27.743581 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:27.743422 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:29.743679 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:29.743649 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:29.744103 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:29.743769 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:29.744103 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:29.743778 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:29.744103 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:29.743818 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:29.744103 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:29.743909 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:29.744103 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:29.744031 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:30.201074 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:30.201038 2874 generic.go:356] "Generic (PLEG): container finished" podID="e9465c98-ea01-4cd6-83b7-485bfb6b54f0" containerID="0a2a890b7bca9dae5e359dbb3113ae4a4823b76b98f9503735ebbdb66a006750" exitCode=0 Sep 18 16:57:30.201232 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:30.201124 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" event={"ID":"e9465c98-ea01-4cd6-83b7-485bfb6b54f0","Type":"ContainerDied","Data":"0a2a890b7bca9dae5e359dbb3113ae4a4823b76b98f9503735ebbdb66a006750"} Sep 18 16:57:30.204463 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:30.204434 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" event={"ID":"74a7c0f8-bcb1-452f-8989-da958fe9ce25","Type":"ContainerStarted","Data":"321e77ca276c404847c69a67408519e305cff0cfc8065adcd1b3483b271b8bbe"} Sep 18 16:57:30.713631 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:30.713597 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:30.713795 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:30.713647 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:30.713795 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:30.713760 2874 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:57:30.713874 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:30.713820 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs podName:bb0c72b0-0d55-4136-99ce-d50036c55d91 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:02.713806302 +0000 UTC m=+256.048170987 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs") pod "network-metrics-daemon-mqrhp" (UID: "bb0c72b0-0d55-4136-99ce-d50036c55d91") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:57:30.713874 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:30.713764 2874 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:57:30.713980 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:30.713882 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret podName:cb442e64-9a04-459c-814a-71a5bf49c5f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:02.713870702 +0000 UTC m=+256.048235387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret") pod "global-pull-secret-syncer-t4fzq" (UID: "cb442e64-9a04-459c-814a-71a5bf49c5f7") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:57:31.209785 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:31.209751 2874 generic.go:356] "Generic (PLEG): container finished" podID="e9465c98-ea01-4cd6-83b7-485bfb6b54f0" containerID="82ce1eb4a350a9ceeff2b35f75e83abf4e0f44494d5eeeeddb37aff48c435e4b" exitCode=0 Sep 18 16:57:31.210321 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:31.209791 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" event={"ID":"e9465c98-ea01-4cd6-83b7-485bfb6b54f0","Type":"ContainerDied","Data":"82ce1eb4a350a9ceeff2b35f75e83abf4e0f44494d5eeeeddb37aff48c435e4b"} Sep 18 16:57:31.743194 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:31.743115 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:31.743325 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:31.743115 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:31.743325 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:31.743248 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4fzq" podUID="cb442e64-9a04-459c-814a-71a5bf49c5f7" Sep 18 16:57:31.743325 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:31.743115 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:31.743465 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:31.743321 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mqrhp" podUID="bb0c72b0-0d55-4136-99ce-d50036c55d91" Sep 18 16:57:31.743465 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:31.743382 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-npjd7" podUID="e9379c30-e23f-4635-a9c1-73d866893686" Sep 18 16:57:32.215752 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:32.215717 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" event={"ID":"e9465c98-ea01-4cd6-83b7-485bfb6b54f0","Type":"ContainerStarted","Data":"6921283e1cf7472114b11a0abee14411a1e36cc4aa7df55db5538b60d5f004bc"} Sep 18 16:57:32.219184 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:32.219160 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" event={"ID":"74a7c0f8-bcb1-452f-8989-da958fe9ce25","Type":"ContainerStarted","Data":"b0f60dbb9946c08250ad2de09dc38e40b88cf95bd200d51961d5589d9fbe01f1"} Sep 18 16:57:32.219439 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:32.219416 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:32.219533 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:32.219485 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:32.219589 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:32.219533 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:32.235108 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:32.235087 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:32.235230 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:32.235181 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:57:32.243508 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:32.243470 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bl7jm" podStartSLOduration=10.274190969 podStartE2EDuration="37.243450717s" totalImagesPullingTime="26.969259748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:32.243097938 +0000 UTC m=+225.577462667" watchObservedRunningTime="2026-09-18 16:57:32.243450717 +0000 UTC m=+225.577815435" Sep 18 16:57:32.264987 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:32.264947 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" podStartSLOduration=7.264934582 podStartE2EDuration="7.264934582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:32.264357449 +0000 UTC m=+225.598722156" watchObservedRunningTime="2026-09-18 16:57:32.264934582 +0000 UTC m=+225.599299289" Sep 18 16:57:33.330063 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:33.330023 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dcrn\" (UniqueName: \"kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn\") pod \"network-check-target-npjd7\" (UID: \"e9379c30-e23f-4635-a9c1-73d866893686\") " pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:33.330717 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:33.330174 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:57:33.330717 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:33.330197 2874 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:57:33.330717 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:33.330211 2874 projected.go:196] Error preparing data for projected volume kube-api-access-9dcrn for pod openshift-network-diagnostics/network-check-target-npjd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:33.330717 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:33.330269 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn podName:e9379c30-e23f-4635-a9c1-73d866893686 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:05.330250868 +0000 UTC m=+258.664615569 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9dcrn" (UniqueName: "kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn") pod "network-check-target-npjd7" (UID: "e9379c30-e23f-4635-a9c1-73d866893686") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:57:33.743193 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:33.743161 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:57:33.743335 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:33.743202 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:57:33.743335 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:33.743203 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:57:33.746303 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:33.746286 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d4kk4\"" Sep 18 16:57:33.746386 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:33.746286 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 16:57:33.747285 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:33.747269 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 16:57:33.747334 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:33.747300 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 16:57:33.747334 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:33.747310 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 16:57:33.747440 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:33.747345 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-plhlv\"" Sep 18 16:57:39.177765 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.177737 2874 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-99.ec2.internal" event="NodeReady" Sep 18 16:57:39.219321 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.219293 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-v5qkh"] Sep 18 16:57:39.224153 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.224132 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd"] Sep 18 16:57:39.224285 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.224233 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.227145 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.227125 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 18 16:57:39.228067 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.228046 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7ccfc6f4f8-ksblq"] Sep 18 16:57:39.228240 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.228220 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:39.231950 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.231928 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 18 16:57:39.232058 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.232008 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 18 16:57:39.233000 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.232954 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 18 16:57:39.233101 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.233082 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:57:39.233149 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.233140 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tq7q4\"" Sep 18 16:57:39.233794 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.233779 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:57:39.233889 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.233838 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 18 16:57:39.235828 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.235814 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7"] Sep 18 16:57:39.235931 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.235910 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.239075 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.239059 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 18 16:57:39.239242 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.239224 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9wmn4\"" Sep 18 16:57:39.239570 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.239552 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 18 16:57:39.240970 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.240952 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 18 16:57:39.241067 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.240990 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 18 16:57:39.241773 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.241755 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 18 16:57:39.242071 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.242039 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 18 16:57:39.243248 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.243231 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dtp4g\"" Sep 18 16:57:39.251008 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.250965 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 18 16:57:39.253806 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.253785 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6"] Sep 18 16:57:39.253939 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.253922 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.258495 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.258476 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:57:39.258630 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.258606 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ppc2f\"" Sep 18 16:57:39.259010 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.258989 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 18 16:57:39.263894 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.263871 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 18 16:57:39.264023 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.263878 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 18 16:57:39.265647 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.265624 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.265746 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.265669 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.265797 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.265741 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:39.265797 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.265764 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-stats-auth\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.265874 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.265797 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-default-certificate\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.265874 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.265821 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-serving-cert\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.265874 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.265849 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-config\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.265874 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.265866 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-trusted-ca\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.266038 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.265884 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sf7b\" (UniqueName: \"kubernetes.io/projected/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-kube-api-access-7sf7b\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.266038 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.265941 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n4nq\" (UniqueName: \"kubernetes.io/projected/0c2d3679-f1c1-4769-9266-bcee43b95806-kube-api-access-4n4nq\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:39.266110 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.266036 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v569g\" (UniqueName: \"kubernetes.io/projected/7897222e-78f1-4920-9afe-669e89dffcab-kube-api-access-v569g\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.276097 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.276078 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc"] Sep 18 16:57:39.276208 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.276183 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:39.284000 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.283983 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 18 16:57:39.284091 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.283989 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 18 16:57:39.284349 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.284331 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-djfp2\"" Sep 18 16:57:39.284430 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.284415 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 18 16:57:39.284492 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.284457 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 18 16:57:39.300070 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.300051 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56b5b5c5f4-tpsp6"] Sep 18 16:57:39.300198 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.300182 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc" Sep 18 16:57:39.303597 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.303581 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jpndk\"" Sep 18 16:57:39.320009 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.319985 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm"] Sep 18 16:57:39.320106 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.320091 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.332131 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.332116 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-phlgr"] Sep 18 16:57:39.332230 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.332217 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:39.336746 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.336719 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fmtn2\"" Sep 18 16:57:39.336866 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.336849 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 18 16:57:39.336953 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.336941 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 18 16:57:39.337032 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.336977 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 18 16:57:39.337032 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.337023 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 18 16:57:39.343092 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.343075 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 18 16:57:39.343172 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.343077 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 18 16:57:39.343172 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.343118 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-m5qs4\"" Sep 18 16:57:39.354201 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.354161 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8"] Sep 18 16:57:39.354287 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.354237 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.367156 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367139 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.367255 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367164 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59db72b3-cbc6-4634-bdcc-970021914be1-serving-cert\") pod \"service-ca-operator-57cdc4b768-4n4f7\" (UID: \"59db72b3-cbc6-4634-bdcc-970021914be1\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.367255 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367199 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:39.367255 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367219 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-stats-auth\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.367440 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.367268 2874 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:57:39.367440 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367280 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-default-certificate\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.367440 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.367308 2874 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:57:39.367440 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.367321 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:57:39.867306877 +0000 UTC m=+233.201671565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : secret "router-metrics-certs-default" not found Sep 18 16:57:39.367440 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367307 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-serving-cert\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.367440 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.367364 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls podName:0c2d3679-f1c1-4769-9266-bcee43b95806 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:39.86734596 +0000 UTC m=+233.201710664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b5ljd" (UID: "0c2d3679-f1c1-4769-9266-bcee43b95806") : secret "samples-operator-tls" not found Sep 18 16:57:39.367440 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367415 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59db72b3-cbc6-4634-bdcc-970021914be1-config\") pod \"service-ca-operator-57cdc4b768-4n4f7\" (UID: \"59db72b3-cbc6-4634-bdcc-970021914be1\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367445 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2bsd\" (UniqueName: \"kubernetes.io/projected/0b69a460-ede3-419c-ae98-8569c701ef76-kube-api-access-d2bsd\") pod \"network-check-source-7cbc45fddc-j9zqc\" (UID: \"0b69a460-ede3-419c-ae98-8569c701ef76\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367472 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4qsc\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-kube-api-access-h4qsc\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367498 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367535 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-config\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367600 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-trusted-ca\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367628 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rvh\" (UniqueName: \"kubernetes.io/projected/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-kube-api-access-f5rvh\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367655 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-certificates\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367680 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-trusted-ca\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367704 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367738 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sf7b\" (UniqueName: \"kubernetes.io/projected/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-kube-api-access-7sf7b\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367771 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n4nq\" (UniqueName: \"kubernetes.io/projected/0c2d3679-f1c1-4769-9266-bcee43b95806-kube-api-access-4n4nq\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367796 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367825 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvzs\" (UniqueName: \"kubernetes.io/projected/59db72b3-cbc6-4634-bdcc-970021914be1-kube-api-access-zcvzs\") pod \"service-ca-operator-57cdc4b768-4n4f7\" (UID: \"59db72b3-cbc6-4634-bdcc-970021914be1\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.367871 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367848 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-bound-sa-token\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.368793 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367879 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-image-registry-private-configuration\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.368793 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367906 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-ca-trust-extracted\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.368793 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367946 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v569g\" (UniqueName: \"kubernetes.io/projected/7897222e-78f1-4920-9afe-669e89dffcab-kube-api-access-v569g\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.368793 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.367973 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:39.368793 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.368002 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:39.368793 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.368033 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.368793 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.368057 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-installation-pull-secrets\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.368793 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.368524 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-trusted-ca\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.369358 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.368841 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:57:39.868828307 +0000 UTC m=+233.203192992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : configmap references non-existent config key: service-ca.crt Sep 18 16:57:39.370022 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.370002 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-config\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.370775 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.370755 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-default-certificate\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.371203 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.371181 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-stats-auth\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.371483 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.371461 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-serving-cert\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.372502 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.372484 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 18 16:57:39.372808 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.372762 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 18 16:57:39.372908 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.372682 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 18 16:57:39.373352 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.373332 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 18 16:57:39.373467 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.373335 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l59ld\"" Sep 18 16:57:39.378337 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.378320 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6"] Sep 18 16:57:39.378464 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.378449 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8" Sep 18 16:57:39.381335 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.381251 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-h84ms\"" Sep 18 16:57:39.384661 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.384306 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:57:39.384661 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.384334 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 18 16:57:39.385609 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.385589 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 18 16:57:39.393550 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.393531 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sf7b\" (UniqueName: \"kubernetes.io/projected/0d14e37c-3b2e-4192-8e48-8cc1bfba3707-kube-api-access-7sf7b\") pod \"console-operator-fd4bb46c-v5qkh\" (UID: \"0d14e37c-3b2e-4192-8e48-8cc1bfba3707\") " pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.394375 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.394360 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n4nq\" (UniqueName: \"kubernetes.io/projected/0c2d3679-f1c1-4769-9266-bcee43b95806-kube-api-access-4n4nq\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:39.394953 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.394939 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v569g\" (UniqueName: \"kubernetes.io/projected/7897222e-78f1-4920-9afe-669e89dffcab-kube-api-access-v569g\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.396944 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.396925 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-v5qkh"] Sep 18 16:57:39.397004 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.396959 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6"] Sep 18 16:57:39.397004 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.396982 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc"] Sep 18 16:57:39.397004 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.397001 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6"] Sep 18 16:57:39.397106 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.397014 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.397149 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.397019 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7ccfc6f4f8-ksblq"] Sep 18 16:57:39.397461 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.397160 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-qqdzb"] Sep 18 16:57:39.400077 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.400057 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 18 16:57:39.400207 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.400121 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:57:39.400207 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.400129 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7j474\"" Sep 18 16:57:39.400207 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.400153 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 18 16:57:39.400207 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.400125 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 18 16:57:39.412738 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.412721 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-phlgr"] Sep 18 16:57:39.412815 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.412748 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm"] Sep 18 16:57:39.412815 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.412766 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8"] Sep 18 16:57:39.412815 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.412795 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bdv7m"] Sep 18 16:57:39.412908 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.412827 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.415199 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.415182 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\"" Sep 18 16:57:39.434839 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.434796 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd"] Sep 18 16:57:39.434839 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.434825 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7"] Sep 18 16:57:39.434956 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.434842 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bdv7m"] Sep 18 16:57:39.434956 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.434855 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56b5b5c5f4-tpsp6"] Sep 18 16:57:39.434956 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.434872 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gklt9"] Sep 18 16:57:39.434956 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.434903 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bdv7m" Sep 18 16:57:39.437639 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.437622 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 18 16:57:39.437741 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.437686 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hn8sq\"" Sep 18 16:57:39.437917 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.437898 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 18 16:57:39.437917 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.437906 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 18 16:57:39.456865 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.456844 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gklt9"] Sep 18 16:57:39.456942 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.456929 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.460313 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.460296 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kgcdr\"" Sep 18 16:57:39.460646 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.460629 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 18 16:57:39.461442 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.461424 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 18 16:57:39.461921 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.461905 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 18 16:57:39.462302 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.462280 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 18 16:57:39.468349 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468332 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e131f4d-c737-4144-967f-cebe6ed736c1-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.468439 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468361 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-certificates\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.468439 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468379 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-trusted-ca\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.468439 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468418 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvzs\" (UniqueName: \"kubernetes.io/projected/59db72b3-cbc6-4634-bdcc-970021914be1-kube-api-access-zcvzs\") pod \"service-ca-operator-57cdc4b768-4n4f7\" (UID: \"59db72b3-cbc6-4634-bdcc-970021914be1\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.468547 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468459 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-bound-sa-token\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.468547 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468500 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-image-registry-private-configuration\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.468634 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468545 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e131f4d-c737-4144-967f-cebe6ed736c1-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.468634 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468574 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl58v\" (UniqueName: \"kubernetes.io/projected/af9797ce-2519-4a42-82cc-a68378abdb3a-kube-api-access-gl58v\") pod \"volume-data-source-validator-5c4db8bdf9-jzsb8\" (UID: \"af9797ce-2519-4a42-82cc-a68378abdb3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8" Sep 18 16:57:39.468741 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468669 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52cnf\" (UniqueName: \"kubernetes.io/projected/8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0-kube-api-access-52cnf\") pod \"ingress-canary-bdv7m\" (UID: \"8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0\") " pod="openshift-ingress-canary/ingress-canary-bdv7m" Sep 18 16:57:39.468741 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468694 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-installation-pull-secrets\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.468741 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468716 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sb78\" (UniqueName: \"kubernetes.io/projected/1e131f4d-c737-4144-967f-cebe6ed736c1-kube-api-access-5sb78\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.468950 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468751 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59db72b3-cbc6-4634-bdcc-970021914be1-serving-cert\") pod \"service-ca-operator-57cdc4b768-4n4f7\" (UID: \"59db72b3-cbc6-4634-bdcc-970021914be1\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.469016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468951 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59db72b3-cbc6-4634-bdcc-970021914be1-config\") pod \"service-ca-operator-57cdc4b768-4n4f7\" (UID: \"59db72b3-cbc6-4634-bdcc-970021914be1\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.469016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.468984 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2bsd\" (UniqueName: \"kubernetes.io/projected/0b69a460-ede3-419c-ae98-8569c701ef76-kube-api-access-d2bsd\") pod \"network-check-source-7cbc45fddc-j9zqc\" (UID: \"0b69a460-ede3-419c-ae98-8569c701ef76\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc" Sep 18 16:57:39.469016 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469009 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:39.469177 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469034 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea029ff9-8b9e-4e78-82dd-030bce98e106-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.469177 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469069 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rvh\" (UniqueName: \"kubernetes.io/projected/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-kube-api-access-f5rvh\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:39.469177 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469077 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-certificates\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.469177 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469093 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea029ff9-8b9e-4e78-82dd-030bce98e106-snapshots\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.469177 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469141 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hv5\" (UniqueName: \"kubernetes.io/projected/ea029ff9-8b9e-4e78-82dd-030bce98e106-kube-api-access-27hv5\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.469424 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469188 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.469424 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469298 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea029ff9-8b9e-4e78-82dd-030bce98e106-serving-cert\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.469424 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469310 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-trusted-ca\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.469424 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469335 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-ca-trust-extracted\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.469424 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469360 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea029ff9-8b9e-4e78-82dd-030bce98e106-tmp\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.469762 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.469633 2874 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:57:39.469762 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.469648 2874 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b5b5c5f4-tpsp6: secret "image-registry-tls" not found Sep 18 16:57:39.469762 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.469713 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls podName:5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:39.969696376 +0000 UTC m=+233.304061064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls") pod "image-registry-56b5b5c5f4-tpsp6" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1") : secret "image-registry-tls" not found Sep 18 16:57:39.469762 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469731 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-ca-trust-extracted\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.469762 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469751 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59db72b3-cbc6-4634-bdcc-970021914be1-config\") pod \"service-ca-operator-57cdc4b768-4n4f7\" (UID: \"59db72b3-cbc6-4634-bdcc-970021914be1\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.470023 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469823 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:39.470023 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469886 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:39.470023 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469920 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea029ff9-8b9e-4e78-82dd-030bce98e106-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.470023 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469946 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0-cert\") pod \"ingress-canary-bdv7m\" (UID: \"8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0\") " pod="openshift-ingress-canary/ingress-canary-bdv7m" Sep 18 16:57:39.470023 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.469990 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fea808a0-7a31-4e8c-a397-033dec7481ab-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-kqbn6\" (UID: \"fea808a0-7a31-4e8c-a397-033dec7481ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.470023 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.470002 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:39.470023 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.470014 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qvm\" (UniqueName: \"kubernetes.io/projected/fea808a0-7a31-4e8c-a397-033dec7481ab-kube-api-access-k2qvm\") pod \"kube-storage-version-migrator-operator-55889f7b47-kqbn6\" (UID: \"fea808a0-7a31-4e8c-a397-033dec7481ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.470320 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.470006 2874 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:39.470320 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.470052 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/1e131f4d-c737-4144-967f-cebe6ed736c1-ready\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.470320 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.470089 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls podName:a9d00046-58ab-4f9c-b9bd-85b4853f2fd8 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:39.970074438 +0000 UTC m=+233.304439127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7gbc6" (UID: "a9d00046-58ab-4f9c-b9bd-85b4853f2fd8") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:39.470320 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.470142 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fea808a0-7a31-4e8c-a397-033dec7481ab-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-kqbn6\" (UID: \"fea808a0-7a31-4e8c-a397-033dec7481ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.470320 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.470181 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:39.470320 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.470225 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4qsc\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-kube-api-access-h4qsc\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.470320 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.470253 2874 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:57:39.470320 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.470319 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert podName:1de7a1b3-6bc5-4f4c-832a-677bd92764f1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:39.970304593 +0000 UTC m=+233.304669293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-w2jnm" (UID: "1de7a1b3-6bc5-4f4c-832a-677bd92764f1") : secret "networking-console-plugin-cert" not found Sep 18 16:57:39.470692 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.470494 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:39.471266 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.471243 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-image-registry-private-configuration\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.471534 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.471516 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59db72b3-cbc6-4634-bdcc-970021914be1-serving-cert\") pod \"service-ca-operator-57cdc4b768-4n4f7\" (UID: \"59db72b3-cbc6-4634-bdcc-970021914be1\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.471729 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.471682 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-installation-pull-secrets\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.485875 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.485856 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvzs\" (UniqueName: \"kubernetes.io/projected/59db72b3-cbc6-4634-bdcc-970021914be1-kube-api-access-zcvzs\") pod \"service-ca-operator-57cdc4b768-4n4f7\" (UID: \"59db72b3-cbc6-4634-bdcc-970021914be1\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.486207 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.486191 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2bsd\" (UniqueName: \"kubernetes.io/projected/0b69a460-ede3-419c-ae98-8569c701ef76-kube-api-access-d2bsd\") pod \"network-check-source-7cbc45fddc-j9zqc\" (UID: \"0b69a460-ede3-419c-ae98-8569c701ef76\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc" Sep 18 16:57:39.486612 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.486595 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rvh\" (UniqueName: \"kubernetes.io/projected/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-kube-api-access-f5rvh\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:39.486800 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.486781 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4qsc\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-kube-api-access-h4qsc\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.488383 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.488367 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-bound-sa-token\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.534727 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.534693 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:39.564969 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.564953 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" Sep 18 16:57:39.571320 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571242 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hv5\" (UniqueName: \"kubernetes.io/projected/ea029ff9-8b9e-4e78-82dd-030bce98e106-kube-api-access-27hv5\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.571320 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571278 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fab1373-7e52-4716-ae69-44a8d87cc7ae-config-volume\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.571320 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571311 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea029ff9-8b9e-4e78-82dd-030bce98e106-serving-cert\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.571320 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571336 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.571320 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571361 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea029ff9-8b9e-4e78-82dd-030bce98e106-tmp\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.571739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571419 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea029ff9-8b9e-4e78-82dd-030bce98e106-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.571739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571442 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0-cert\") pod \"ingress-canary-bdv7m\" (UID: \"8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0\") " pod="openshift-ingress-canary/ingress-canary-bdv7m" Sep 18 16:57:39.571739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571481 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fea808a0-7a31-4e8c-a397-033dec7481ab-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-kqbn6\" (UID: \"fea808a0-7a31-4e8c-a397-033dec7481ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.571739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571507 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qvm\" (UniqueName: \"kubernetes.io/projected/fea808a0-7a31-4e8c-a397-033dec7481ab-kube-api-access-k2qvm\") pod \"kube-storage-version-migrator-operator-55889f7b47-kqbn6\" (UID: \"fea808a0-7a31-4e8c-a397-033dec7481ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.571739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571531 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/1e131f4d-c737-4144-967f-cebe6ed736c1-ready\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.571739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571565 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fea808a0-7a31-4e8c-a397-033dec7481ab-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-kqbn6\" (UID: \"fea808a0-7a31-4e8c-a397-033dec7481ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.571739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571630 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e131f4d-c737-4144-967f-cebe6ed736c1-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.571739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571689 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e131f4d-c737-4144-967f-cebe6ed736c1-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.571739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571728 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl58v\" (UniqueName: \"kubernetes.io/projected/af9797ce-2519-4a42-82cc-a68378abdb3a-kube-api-access-gl58v\") pod \"volume-data-source-validator-5c4db8bdf9-jzsb8\" (UID: \"af9797ce-2519-4a42-82cc-a68378abdb3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8" Sep 18 16:57:39.572312 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571755 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52cnf\" (UniqueName: \"kubernetes.io/projected/8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0-kube-api-access-52cnf\") pod \"ingress-canary-bdv7m\" (UID: \"8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0\") " pod="openshift-ingress-canary/ingress-canary-bdv7m" Sep 18 16:57:39.572312 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571769 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea029ff9-8b9e-4e78-82dd-030bce98e106-tmp\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.572312 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571783 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fab1373-7e52-4716-ae69-44a8d87cc7ae-tmp-dir\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.572312 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571830 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sb78\" (UniqueName: \"kubernetes.io/projected/1e131f4d-c737-4144-967f-cebe6ed736c1-kube-api-access-5sb78\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.572312 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571922 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea029ff9-8b9e-4e78-82dd-030bce98e106-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.572312 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571954 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea029ff9-8b9e-4e78-82dd-030bce98e106-snapshots\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.572312 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.571981 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg8pk\" (UniqueName: \"kubernetes.io/projected/3fab1373-7e52-4716-ae69-44a8d87cc7ae-kube-api-access-gg8pk\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.572312 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.572135 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fea808a0-7a31-4e8c-a397-033dec7481ab-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-kqbn6\" (UID: \"fea808a0-7a31-4e8c-a397-033dec7481ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.572312 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.572274 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e131f4d-c737-4144-967f-cebe6ed736c1-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.572312 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.572307 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/1e131f4d-c737-4144-967f-cebe6ed736c1-ready\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.572907 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.572322 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea029ff9-8b9e-4e78-82dd-030bce98e106-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.572907 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.572770 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea029ff9-8b9e-4e78-82dd-030bce98e106-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.572907 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.572805 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea029ff9-8b9e-4e78-82dd-030bce98e106-snapshots\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.572907 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.572853 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e131f4d-c737-4144-967f-cebe6ed736c1-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.574022 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.573994 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea029ff9-8b9e-4e78-82dd-030bce98e106-serving-cert\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.574844 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.574826 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fea808a0-7a31-4e8c-a397-033dec7481ab-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-kqbn6\" (UID: \"fea808a0-7a31-4e8c-a397-033dec7481ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.574969 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.574954 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0-cert\") pod \"ingress-canary-bdv7m\" (UID: \"8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0\") " pod="openshift-ingress-canary/ingress-canary-bdv7m" Sep 18 16:57:39.581510 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.581478 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qvm\" (UniqueName: \"kubernetes.io/projected/fea808a0-7a31-4e8c-a397-033dec7481ab-kube-api-access-k2qvm\") pod \"kube-storage-version-migrator-operator-55889f7b47-kqbn6\" (UID: \"fea808a0-7a31-4e8c-a397-033dec7481ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.581642 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.581626 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hv5\" (UniqueName: \"kubernetes.io/projected/ea029ff9-8b9e-4e78-82dd-030bce98e106-kube-api-access-27hv5\") pod \"insights-operator-6f9c644dd9-phlgr\" (UID: \"ea029ff9-8b9e-4e78-82dd-030bce98e106\") " pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.581830 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.581809 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl58v\" (UniqueName: \"kubernetes.io/projected/af9797ce-2519-4a42-82cc-a68378abdb3a-kube-api-access-gl58v\") pod \"volume-data-source-validator-5c4db8bdf9-jzsb8\" (UID: \"af9797ce-2519-4a42-82cc-a68378abdb3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8" Sep 18 16:57:39.581902 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.581875 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sb78\" (UniqueName: \"kubernetes.io/projected/1e131f4d-c737-4144-967f-cebe6ed736c1-kube-api-access-5sb78\") pod \"cni-sysctl-allowlist-ds-qqdzb\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.582413 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.582380 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52cnf\" (UniqueName: \"kubernetes.io/projected/8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0-kube-api-access-52cnf\") pod \"ingress-canary-bdv7m\" (UID: \"8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0\") " pod="openshift-ingress-canary/ingress-canary-bdv7m" Sep 18 16:57:39.608646 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.608621 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc" Sep 18 16:57:39.661982 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.661706 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" Sep 18 16:57:39.673494 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.673463 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fab1373-7e52-4716-ae69-44a8d87cc7ae-tmp-dir\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.673630 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.673562 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg8pk\" (UniqueName: \"kubernetes.io/projected/3fab1373-7e52-4716-ae69-44a8d87cc7ae-kube-api-access-gg8pk\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.673630 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.673597 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fab1373-7e52-4716-ae69-44a8d87cc7ae-config-volume\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.673743 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.673633 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.673794 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.673766 2874 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:57:39.673847 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.673825 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls podName:3fab1373-7e52-4716-ae69-44a8d87cc7ae nodeName:}" failed. No retries permitted until 2026-09-18 16:57:40.173806134 +0000 UTC m=+233.508170839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls") pod "dns-default-gklt9" (UID: "3fab1373-7e52-4716-ae69-44a8d87cc7ae") : secret "dns-default-metrics-tls" not found Sep 18 16:57:39.675224 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.674345 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fab1373-7e52-4716-ae69-44a8d87cc7ae-tmp-dir\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.675224 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.675116 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fab1373-7e52-4716-ae69-44a8d87cc7ae-config-volume\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.685991 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.685940 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg8pk\" (UniqueName: \"kubernetes.io/projected/3fab1373-7e52-4716-ae69-44a8d87cc7ae-kube-api-access-gg8pk\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:39.689797 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.689724 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8" Sep 18 16:57:39.709872 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.709702 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" Sep 18 16:57:39.720090 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.720050 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:39.737715 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.737684 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-l6cpb"] Sep 18 16:57:39.744009 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.743375 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bdv7m" Sep 18 16:57:39.758560 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.757103 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:39.760872 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.760763 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w66jd\"" Sep 18 16:57:39.875702 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.875673 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzdp5\" (UniqueName: \"kubernetes.io/projected/175a17f1-cd97-40d2-b70a-e8ba235c28eb-kube-api-access-nzdp5\") pod \"node-resolver-l6cpb\" (UID: \"175a17f1-cd97-40d2-b70a-e8ba235c28eb\") " pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:39.875840 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.875715 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/175a17f1-cd97-40d2-b70a-e8ba235c28eb-tmp-dir\") pod \"node-resolver-l6cpb\" (UID: \"175a17f1-cd97-40d2-b70a-e8ba235c28eb\") " pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:39.875840 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.875770 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/175a17f1-cd97-40d2-b70a-e8ba235c28eb-hosts-file\") pod \"node-resolver-l6cpb\" (UID: \"175a17f1-cd97-40d2-b70a-e8ba235c28eb\") " pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:39.875840 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.875821 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.876008 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.875876 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:39.876008 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.875949 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:39.876008 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.875984 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:57:40.87596109 +0000 UTC m=+234.210325783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : configmap references non-existent config key: service-ca.crt Sep 18 16:57:39.876163 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.876022 2874 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:57:39.876163 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.876067 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls podName:0c2d3679-f1c1-4769-9266-bcee43b95806 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:40.876055051 +0000 UTC m=+234.210419750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b5ljd" (UID: "0c2d3679-f1c1-4769-9266-bcee43b95806") : secret "samples-operator-tls" not found Sep 18 16:57:39.876163 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.876070 2874 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:57:39.876163 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.876124 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:57:40.876107349 +0000 UTC m=+234.210472052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : secret "router-metrics-certs-default" not found Sep 18 16:57:39.887883 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.887856 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7"] Sep 18 16:57:39.890343 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:39.890312 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59db72b3_cbc6_4634_bdcc_970021914be1.slice/crio-27d3df8c34a923273049a71adf523a7bb5ee03c646c0547055378f3d1f26e938 WatchSource:0}: Error finding container 27d3df8c34a923273049a71adf523a7bb5ee03c646c0547055378f3d1f26e938: Status 404 returned error can't find the container with id 27d3df8c34a923273049a71adf523a7bb5ee03c646c0547055378f3d1f26e938 Sep 18 16:57:39.893226 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.893207 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-v5qkh"] Sep 18 16:57:39.895570 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:39.895546 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d14e37c_3b2e_4192_8e48_8cc1bfba3707.slice/crio-251f4e74b8460163c3322b51ed0a6bb87e914f9e216d484eb470a5175f9a5e94 WatchSource:0}: Error finding container 251f4e74b8460163c3322b51ed0a6bb87e914f9e216d484eb470a5175f9a5e94: Status 404 returned error can't find the container with id 251f4e74b8460163c3322b51ed0a6bb87e914f9e216d484eb470a5175f9a5e94 Sep 18 16:57:39.913811 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.913730 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc"] Sep 18 16:57:39.915927 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:39.915896 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b69a460_ede3_419c_ae98_8569c701ef76.slice/crio-4e54ecfc34a561c791cbf5325fb81a81a2af7bb32e697d1e4283b056fd53b3e9 WatchSource:0}: Error finding container 4e54ecfc34a561c791cbf5325fb81a81a2af7bb32e697d1e4283b056fd53b3e9: Status 404 returned error can't find the container with id 4e54ecfc34a561c791cbf5325fb81a81a2af7bb32e697d1e4283b056fd53b3e9 Sep 18 16:57:39.969917 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.969889 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-phlgr"] Sep 18 16:57:39.972095 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:39.972073 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea029ff9_8b9e_4e78_82dd_030bce98e106.slice/crio-422c2f5121b8fe064e0ba15e0dad68b3e5f209e1bdaf7ab7821648f045cfc4f6 WatchSource:0}: Error finding container 422c2f5121b8fe064e0ba15e0dad68b3e5f209e1bdaf7ab7821648f045cfc4f6: Status 404 returned error can't find the container with id 422c2f5121b8fe064e0ba15e0dad68b3e5f209e1bdaf7ab7821648f045cfc4f6 Sep 18 16:57:39.976620 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.976602 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:39.976702 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.976689 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdp5\" (UniqueName: \"kubernetes.io/projected/175a17f1-cd97-40d2-b70a-e8ba235c28eb-kube-api-access-nzdp5\") pod \"node-resolver-l6cpb\" (UID: \"175a17f1-cd97-40d2-b70a-e8ba235c28eb\") " pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:39.976742 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.976710 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/175a17f1-cd97-40d2-b70a-e8ba235c28eb-tmp-dir\") pod \"node-resolver-l6cpb\" (UID: \"175a17f1-cd97-40d2-b70a-e8ba235c28eb\") " pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:39.976742 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.976730 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:39.976854 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.976740 2874 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:57:39.976854 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.976753 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/175a17f1-cd97-40d2-b70a-e8ba235c28eb-hosts-file\") pod \"node-resolver-l6cpb\" (UID: \"175a17f1-cd97-40d2-b70a-e8ba235c28eb\") " pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:39.976854 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.976772 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:39.976854 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.976793 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert podName:1de7a1b3-6bc5-4f4c-832a-677bd92764f1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:40.976774583 +0000 UTC m=+234.311139273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-w2jnm" (UID: "1de7a1b3-6bc5-4f4c-832a-677bd92764f1") : secret "networking-console-plugin-cert" not found Sep 18 16:57:39.976854 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.976839 2874 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:39.977072 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.976857 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/175a17f1-cd97-40d2-b70a-e8ba235c28eb-hosts-file\") pod \"node-resolver-l6cpb\" (UID: \"175a17f1-cd97-40d2-b70a-e8ba235c28eb\") " pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:39.977072 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.976896 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls podName:a9d00046-58ab-4f9c-b9bd-85b4853f2fd8 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:40.976882869 +0000 UTC m=+234.311247554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7gbc6" (UID: "a9d00046-58ab-4f9c-b9bd-85b4853f2fd8") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:39.977072 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.976907 2874 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:57:39.977072 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.976926 2874 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b5b5c5f4-tpsp6: secret "image-registry-tls" not found Sep 18 16:57:39.977072 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:39.976994 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls podName:5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:40.976975986 +0000 UTC m=+234.311340671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls") pod "image-registry-56b5b5c5f4-tpsp6" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1") : secret "image-registry-tls" not found Sep 18 16:57:39.977230 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.977076 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/175a17f1-cd97-40d2-b70a-e8ba235c28eb-tmp-dir\") pod \"node-resolver-l6cpb\" (UID: \"175a17f1-cd97-40d2-b70a-e8ba235c28eb\") " pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:39.982348 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.982324 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8"] Sep 18 16:57:39.984525 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:39.984508 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9797ce_2519_4a42_82cc_a68378abdb3a.slice/crio-a3dc0780dddedac2d8ef99afb8221a43da463fc76bae417ae40fbc832a1fb7c2 WatchSource:0}: Error finding container a3dc0780dddedac2d8ef99afb8221a43da463fc76bae417ae40fbc832a1fb7c2: Status 404 returned error can't find the container with id a3dc0780dddedac2d8ef99afb8221a43da463fc76bae417ae40fbc832a1fb7c2 Sep 18 16:57:39.988371 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:39.988354 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdp5\" (UniqueName: \"kubernetes.io/projected/175a17f1-cd97-40d2-b70a-e8ba235c28eb-kube-api-access-nzdp5\") pod \"node-resolver-l6cpb\" (UID: \"175a17f1-cd97-40d2-b70a-e8ba235c28eb\") " pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:40.014136 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.014112 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6"] Sep 18 16:57:40.016451 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:40.016432 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea808a0_7a31_4e8c_a397_033dec7481ab.slice/crio-6e47a0756c41282d11c13384693d6ea2b1d342156705d99e44b1184bd61b46ce WatchSource:0}: Error finding container 6e47a0756c41282d11c13384693d6ea2b1d342156705d99e44b1184bd61b46ce: Status 404 returned error can't find the container with id 6e47a0756c41282d11c13384693d6ea2b1d342156705d99e44b1184bd61b46ce Sep 18 16:57:40.053652 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.053620 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bdv7m"] Sep 18 16:57:40.056358 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:40.056330 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8055f24c_aa2d_4eb8_adfd_7f8ce04c9bb0.slice/crio-a46e11b33fa3f767340a9e51eeb030147df690fffd8381af5cfe92d9830269ab WatchSource:0}: Error finding container a46e11b33fa3f767340a9e51eeb030147df690fffd8381af5cfe92d9830269ab: Status 404 returned error can't find the container with id a46e11b33fa3f767340a9e51eeb030147df690fffd8381af5cfe92d9830269ab Sep 18 16:57:40.072190 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.072170 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l6cpb" Sep 18 16:57:40.100068 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:57:40.100038 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175a17f1_cd97_40d2_b70a_e8ba235c28eb.slice/crio-b20fce069dab83db9cbef09653b8a963a12965653324961becc58a50389ae093 WatchSource:0}: Error finding container b20fce069dab83db9cbef09653b8a963a12965653324961becc58a50389ae093: Status 404 returned error can't find the container with id b20fce069dab83db9cbef09653b8a963a12965653324961becc58a50389ae093 Sep 18 16:57:40.178380 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.178352 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:40.178685 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.178503 2874 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:57:40.178685 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.178563 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls podName:3fab1373-7e52-4716-ae69-44a8d87cc7ae nodeName:}" failed. No retries permitted until 2026-09-18 16:57:41.178548608 +0000 UTC m=+234.512913296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls") pod "dns-default-gklt9" (UID: "3fab1373-7e52-4716-ae69-44a8d87cc7ae") : secret "dns-default-metrics-tls" not found Sep 18 16:57:40.238727 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.238694 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l6cpb" event={"ID":"175a17f1-cd97-40d2-b70a-e8ba235c28eb","Type":"ContainerStarted","Data":"b20fce069dab83db9cbef09653b8a963a12965653324961becc58a50389ae093"} Sep 18 16:57:40.239860 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.239834 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" event={"ID":"59db72b3-cbc6-4634-bdcc-970021914be1","Type":"ContainerStarted","Data":"27d3df8c34a923273049a71adf523a7bb5ee03c646c0547055378f3d1f26e938"} Sep 18 16:57:40.241218 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.241197 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" event={"ID":"1e131f4d-c737-4144-967f-cebe6ed736c1","Type":"ContainerStarted","Data":"4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea"} Sep 18 16:57:40.241287 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.241235 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" event={"ID":"1e131f4d-c737-4144-967f-cebe6ed736c1","Type":"ContainerStarted","Data":"bc74940a714ff1faba43535f8dc7c9acf53c6c5f3e6f46ac218abb4fefa52fb4"} Sep 18 16:57:40.241511 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.241481 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:40.242354 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.242332 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8" event={"ID":"af9797ce-2519-4a42-82cc-a68378abdb3a","Type":"ContainerStarted","Data":"a3dc0780dddedac2d8ef99afb8221a43da463fc76bae417ae40fbc832a1fb7c2"} Sep 18 16:57:40.243352 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.243335 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" event={"ID":"ea029ff9-8b9e-4e78-82dd-030bce98e106","Type":"ContainerStarted","Data":"422c2f5121b8fe064e0ba15e0dad68b3e5f209e1bdaf7ab7821648f045cfc4f6"} Sep 18 16:57:40.244516 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.244493 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc" event={"ID":"0b69a460-ede3-419c-ae98-8569c701ef76","Type":"ContainerStarted","Data":"4e54ecfc34a561c791cbf5325fb81a81a2af7bb32e697d1e4283b056fd53b3e9"} Sep 18 16:57:40.245706 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.245681 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" event={"ID":"0d14e37c-3b2e-4192-8e48-8cc1bfba3707","Type":"ContainerStarted","Data":"251f4e74b8460163c3322b51ed0a6bb87e914f9e216d484eb470a5175f9a5e94"} Sep 18 16:57:40.246928 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.246903 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" event={"ID":"fea808a0-7a31-4e8c-a397-033dec7481ab","Type":"ContainerStarted","Data":"6e47a0756c41282d11c13384693d6ea2b1d342156705d99e44b1184bd61b46ce"} Sep 18 16:57:40.248217 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.248198 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bdv7m" event={"ID":"8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0","Type":"ContainerStarted","Data":"a46e11b33fa3f767340a9e51eeb030147df690fffd8381af5cfe92d9830269ab"} Sep 18 16:57:40.251520 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.251503 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:57:40.256950 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.256911 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" podStartSLOduration=1.256898064 podStartE2EDuration="1.256898064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:40.256088057 +0000 UTC m=+233.590452767" watchObservedRunningTime="2026-09-18 16:57:40.256898064 +0000 UTC m=+233.591262771" Sep 18 16:57:40.885723 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.884736 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:40.885723 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.884831 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:40.885723 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.884954 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:40.885723 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.885106 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:57:42.88508635 +0000 UTC m=+236.219451048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : configmap references non-existent config key: service-ca.crt Sep 18 16:57:40.885723 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.885538 2874 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:57:40.885723 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.885593 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls podName:0c2d3679-f1c1-4769-9266-bcee43b95806 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:42.885577063 +0000 UTC m=+236.219941752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b5ljd" (UID: "0c2d3679-f1c1-4769-9266-bcee43b95806") : secret "samples-operator-tls" not found Sep 18 16:57:40.885723 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.885652 2874 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:57:40.885723 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.885687 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:57:42.885675597 +0000 UTC m=+236.220040294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : secret "router-metrics-certs-default" not found Sep 18 16:57:40.985958 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.985834 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:40.985958 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.985903 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:40.986189 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.986021 2874 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:40.986189 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.986079 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls podName:a9d00046-58ab-4f9c-b9bd-85b4853f2fd8 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:42.986062926 +0000 UTC m=+236.320427613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7gbc6" (UID: "a9d00046-58ab-4f9c-b9bd-85b4853f2fd8") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:40.986189 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.986096 2874 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:57:40.986189 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.986119 2874 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b5b5c5f4-tpsp6: secret "image-registry-tls" not found Sep 18 16:57:40.986189 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.986189 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls podName:5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:42.986156009 +0000 UTC m=+236.320520705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls") pod "image-registry-56b5b5c5f4-tpsp6" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1") : secret "image-registry-tls" not found Sep 18 16:57:40.986423 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:40.986254 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:40.986423 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.986374 2874 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:57:40.986500 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:40.986451 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert podName:1de7a1b3-6bc5-4f4c-832a-677bd92764f1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:42.986438064 +0000 UTC m=+236.320802772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-w2jnm" (UID: "1de7a1b3-6bc5-4f4c-832a-677bd92764f1") : secret "networking-console-plugin-cert" not found Sep 18 16:57:41.188586 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:41.188549 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:41.189053 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:41.188716 2874 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:57:41.189053 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:41.188779 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls podName:3fab1373-7e52-4716-ae69-44a8d87cc7ae nodeName:}" failed. No retries permitted until 2026-09-18 16:57:43.188760651 +0000 UTC m=+236.523125354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls") pod "dns-default-gklt9" (UID: "3fab1373-7e52-4716-ae69-44a8d87cc7ae") : secret "dns-default-metrics-tls" not found Sep 18 16:57:41.259214 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:41.259052 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l6cpb" event={"ID":"175a17f1-cd97-40d2-b70a-e8ba235c28eb","Type":"ContainerStarted","Data":"a3636ae552e2265fc4221895875c45a658259b9acf8d052746e38dde424577f5"} Sep 18 16:57:41.776088 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:41.775115 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l6cpb" podStartSLOduration=2.775092609 podStartE2EDuration="2.775092609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:41.273926192 +0000 UTC m=+234.608290899" watchObservedRunningTime="2026-09-18 16:57:41.775092609 +0000 UTC m=+235.109457317" Sep 18 16:57:41.776088 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:41.775901 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-qqdzb"] Sep 18 16:57:42.907745 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:42.907701 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:42.908308 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:42.907767 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:42.908308 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:42.907888 2874 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:57:42.908308 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:42.907912 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:57:46.907889798 +0000 UTC m=+240.242254506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : configmap references non-existent config key: service-ca.crt Sep 18 16:57:42.908308 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:42.907946 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls podName:0c2d3679-f1c1-4769-9266-bcee43b95806 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:46.907930208 +0000 UTC m=+240.242294913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b5ljd" (UID: "0c2d3679-f1c1-4769-9266-bcee43b95806") : secret "samples-operator-tls" not found Sep 18 16:57:42.908308 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:42.908004 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:42.908308 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:42.908170 2874 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:57:42.908308 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:42.908229 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:57:46.908212946 +0000 UTC m=+240.242577644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : secret "router-metrics-certs-default" not found Sep 18 16:57:43.009039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:43.008952 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:43.009039 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:43.009022 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:43.009254 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:43.009068 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:43.009254 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:43.009131 2874 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:57:43.009254 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:43.009159 2874 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b5b5c5f4-tpsp6: secret "image-registry-tls" not found Sep 18 16:57:43.009254 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:43.009196 2874 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:43.009254 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:43.009212 2874 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:57:43.009254 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:43.009221 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls podName:5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:47.009204473 +0000 UTC m=+240.343569161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls") pod "image-registry-56b5b5c5f4-tpsp6" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1") : secret "image-registry-tls" not found Sep 18 16:57:43.009254 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:43.009255 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert podName:1de7a1b3-6bc5-4f4c-832a-677bd92764f1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:47.009238334 +0000 UTC m=+240.343603027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-w2jnm" (UID: "1de7a1b3-6bc5-4f4c-832a-677bd92764f1") : secret "networking-console-plugin-cert" not found Sep 18 16:57:43.009568 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:43.009274 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls podName:a9d00046-58ab-4f9c-b9bd-85b4853f2fd8 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:47.009264942 +0000 UTC m=+240.343629636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7gbc6" (UID: "a9d00046-58ab-4f9c-b9bd-85b4853f2fd8") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:43.211803 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:43.211761 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:43.211972 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:43.211899 2874 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:57:43.212035 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:43.211976 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls podName:3fab1373-7e52-4716-ae69-44a8d87cc7ae nodeName:}" failed. No retries permitted until 2026-09-18 16:57:47.21195752 +0000 UTC m=+240.546322206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls") pod "dns-default-gklt9" (UID: "3fab1373-7e52-4716-ae69-44a8d87cc7ae") : secret "dns-default-metrics-tls" not found Sep 18 16:57:43.264355 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:43.264257 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" podUID="1e131f4d-c737-4144-967f-cebe6ed736c1" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" gracePeriod=30 Sep 18 16:57:46.945858 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:46.945818 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:46.946326 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:46.945925 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:46.946326 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:46.945983 2874 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:57:46.946326 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:46.946029 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:46.946326 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:46.946057 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls podName:0c2d3679-f1c1-4769-9266-bcee43b95806 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:54.946039815 +0000 UTC m=+248.280404525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b5ljd" (UID: "0c2d3679-f1c1-4769-9266-bcee43b95806") : secret "samples-operator-tls" not found Sep 18 16:57:46.946326 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:46.946083 2874 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:57:46.946326 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:46.946122 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:57:54.946110722 +0000 UTC m=+248.280475411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : configmap references non-existent config key: service-ca.crt Sep 18 16:57:46.946326 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:46.946165 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:57:54.946148006 +0000 UTC m=+248.280512691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : secret "router-metrics-certs-default" not found Sep 18 16:57:47.047417 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:47.047311 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:47.047580 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:47.047419 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:47.047580 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:47.047470 2874 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:47.047580 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:47.047550 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls podName:a9d00046-58ab-4f9c-b9bd-85b4853f2fd8 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:55.047528887 +0000 UTC m=+248.381893577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7gbc6" (UID: "a9d00046-58ab-4f9c-b9bd-85b4853f2fd8") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:47.047580 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:47.047559 2874 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:57:47.047748 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:47.047582 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:47.047748 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:47.047606 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert podName:1de7a1b3-6bc5-4f4c-832a-677bd92764f1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:55.047594706 +0000 UTC m=+248.381959394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-w2jnm" (UID: "1de7a1b3-6bc5-4f4c-832a-677bd92764f1") : secret "networking-console-plugin-cert" not found Sep 18 16:57:47.047748 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:47.047639 2874 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:57:47.047748 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:47.047647 2874 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b5b5c5f4-tpsp6: secret "image-registry-tls" not found Sep 18 16:57:47.047748 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:47.047676 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls podName:5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:55.047668576 +0000 UTC m=+248.382033264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls") pod "image-registry-56b5b5c5f4-tpsp6" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1") : secret "image-registry-tls" not found Sep 18 16:57:47.249319 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:47.249280 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:47.249482 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:47.249447 2874 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:57:47.249523 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:47.249509 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls podName:3fab1373-7e52-4716-ae69-44a8d87cc7ae nodeName:}" failed. No retries permitted until 2026-09-18 16:57:55.249494923 +0000 UTC m=+248.583859609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls") pod "dns-default-gklt9" (UID: "3fab1373-7e52-4716-ae69-44a8d87cc7ae") : secret "dns-default-metrics-tls" not found Sep 18 16:57:48.280597 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.280096 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bdv7m" event={"ID":"8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0","Type":"ContainerStarted","Data":"76be56766e985c7719cd03d9f54075cda18bee410df74cb39dab63d9197b681e"} Sep 18 16:57:48.281743 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.281714 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" event={"ID":"59db72b3-cbc6-4634-bdcc-970021914be1","Type":"ContainerStarted","Data":"fd0eb86514e8d952072be8c24447304dcb60fc585e94c3c60d3afde8585dec43"} Sep 18 16:57:48.283446 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.283393 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8" event={"ID":"af9797ce-2519-4a42-82cc-a68378abdb3a","Type":"ContainerStarted","Data":"c19285de99301f51aeacf1d5c6f1f0956add3de99aac3dd63f244466d674f625"} Sep 18 16:57:48.285058 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.285028 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" event={"ID":"ea029ff9-8b9e-4e78-82dd-030bce98e106","Type":"ContainerStarted","Data":"6fd64f230fc6196b2529eb002774822cd54bbd5f5e9d316e271130883bde1ab2"} Sep 18 16:57:48.286723 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.286699 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc" event={"ID":"0b69a460-ede3-419c-ae98-8569c701ef76","Type":"ContainerStarted","Data":"8b48425b7cc946ec4a9760b69c6758e1fec4a9e46aaf291b4e15926d6cd101f1"} Sep 18 16:57:48.288733 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.288712 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/0.log" Sep 18 16:57:48.288843 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.288753 2874 generic.go:356] "Generic (PLEG): container finished" podID="0d14e37c-3b2e-4192-8e48-8cc1bfba3707" containerID="35ecf7bb6ebc494bea58eba1021cd2d8dfcf13d2586fd6e389cac9e904bcf26b" exitCode=255 Sep 18 16:57:48.288911 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.288835 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" event={"ID":"0d14e37c-3b2e-4192-8e48-8cc1bfba3707","Type":"ContainerDied","Data":"35ecf7bb6ebc494bea58eba1021cd2d8dfcf13d2586fd6e389cac9e904bcf26b"} Sep 18 16:57:48.289045 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.289028 2874 scope.go:117] "RemoveContainer" containerID="35ecf7bb6ebc494bea58eba1021cd2d8dfcf13d2586fd6e389cac9e904bcf26b" Sep 18 16:57:48.290957 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.290936 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" event={"ID":"fea808a0-7a31-4e8c-a397-033dec7481ab","Type":"ContainerStarted","Data":"c7faa4f7adc711a32acb40b561ecfe05a66397d21a915060e9f2e85896a85142"} Sep 18 16:57:48.298185 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.298116 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bdv7m" podStartSLOduration=1.987953329 podStartE2EDuration="9.298101243s" totalImagesPullingTime="7.310147914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:48.297321926 +0000 UTC m=+241.631686634" watchObservedRunningTime="2026-09-18 16:57:48.298101243 +0000 UTC m=+241.632465950" Sep 18 16:57:48.316847 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.316666 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jzsb8" podStartSLOduration=46.004902085 podStartE2EDuration="53.316648745s" totalImagesPullingTime="7.31174666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:48.315625893 +0000 UTC m=+241.649990601" watchObservedRunningTime="2026-09-18 16:57:48.316648745 +0000 UTC m=+241.651013453" Sep 18 16:57:48.332269 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.332213 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-j9zqc" podStartSLOduration=40.869525384 podStartE2EDuration="48.332194356s" totalImagesPullingTime="7.462668972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:48.331159097 +0000 UTC m=+241.665523789" watchObservedRunningTime="2026-09-18 16:57:48.332194356 +0000 UTC m=+241.666559064" Sep 18 16:57:48.381196 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.379215 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" podStartSLOduration=45.984514167 podStartE2EDuration="53.379195412s" totalImagesPullingTime="7.394681245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:48.357630132 +0000 UTC m=+241.691994840" watchObservedRunningTime="2026-09-18 16:57:48.379195412 +0000 UTC m=+241.713560120" Sep 18 16:57:48.400975 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.400908 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" podStartSLOduration=45.924960812 podStartE2EDuration="53.400887883s" totalImagesPullingTime="7.475927071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:48.400040702 +0000 UTC m=+241.734405437" watchObservedRunningTime="2026-09-18 16:57:48.400887883 +0000 UTC m=+241.735252593" Sep 18 16:57:48.424561 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:48.424507 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" podStartSLOduration=46.073649251 podStartE2EDuration="53.424483875s" totalImagesPullingTime="7.350834624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:48.422582945 +0000 UTC m=+241.756947676" watchObservedRunningTime="2026-09-18 16:57:48.424483875 +0000 UTC m=+241.758848583" Sep 18 16:57:49.267912 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.267879 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gg9g5"] Sep 18 16:57:49.289982 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.289950 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gg9g5"] Sep 18 16:57:49.290336 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.290077 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.294905 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.294878 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bm6wt\"" Sep 18 16:57:49.295204 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.295176 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 18 16:57:49.295348 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.295327 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 18 16:57:49.295641 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.295624 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 16:57:49.296070 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.296056 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/0.log" Sep 18 16:57:49.296167 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.296091 2874 generic.go:356] "Generic (PLEG): container finished" podID="0d14e37c-3b2e-4192-8e48-8cc1bfba3707" containerID="5a1200963adf617cc38f81698af838f063fbbcaff92da93f784625037efbe1fd" exitCode=255 Sep 18 16:57:49.296229 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.296185 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" event={"ID":"0d14e37c-3b2e-4192-8e48-8cc1bfba3707","Type":"ContainerDied","Data":"5a1200963adf617cc38f81698af838f063fbbcaff92da93f784625037efbe1fd"} Sep 18 16:57:49.296295 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.296230 2874 scope.go:117] "RemoveContainer" containerID="35ecf7bb6ebc494bea58eba1021cd2d8dfcf13d2586fd6e389cac9e904bcf26b" Sep 18 16:57:49.296623 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.296507 2874 scope.go:117] "RemoveContainer" containerID="5a1200963adf617cc38f81698af838f063fbbcaff92da93f784625037efbe1fd" Sep 18 16:57:49.296728 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:49.296668 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-v5qkh_openshift-console-operator(0d14e37c-3b2e-4192-8e48-8cc1bfba3707)\"" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" podUID="0d14e37c-3b2e-4192-8e48-8cc1bfba3707" Sep 18 16:57:49.371087 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.371051 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.371265 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.371236 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/305b0ad5-11e3-4960-9666-7590e3f2e492-crio-socket\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.371365 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.371344 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkm5v\" (UniqueName: \"kubernetes.io/projected/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-api-access-wkm5v\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.371617 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.371591 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.371780 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.371759 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/305b0ad5-11e3-4960-9666-7590e3f2e492-data-volume\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.473076 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.473045 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.473233 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.473138 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/305b0ad5-11e3-4960-9666-7590e3f2e492-data-volume\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.473233 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:49.473145 2874 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:57:49.473233 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.473187 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.473233 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:49.473212 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls podName:305b0ad5-11e3-4960-9666-7590e3f2e492 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:49.97319237 +0000 UTC m=+243.307557077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gg9g5" (UID: "305b0ad5-11e3-4960-9666-7590e3f2e492") : secret "insights-runtime-extractor-tls" not found Sep 18 16:57:49.473577 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:49.473240 2874 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:57:49.473577 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:49.473292 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm podName:305b0ad5-11e3-4960-9666-7590e3f2e492 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:49.973275023 +0000 UTC m=+243.307639709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gg9g5" (UID: "305b0ad5-11e3-4960-9666-7590e3f2e492") : configmap "kube-rbac-proxy" not found Sep 18 16:57:49.473577 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.473327 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/305b0ad5-11e3-4960-9666-7590e3f2e492-crio-socket\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.473577 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.473392 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkm5v\" (UniqueName: \"kubernetes.io/projected/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-api-access-wkm5v\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.473577 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.473545 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/305b0ad5-11e3-4960-9666-7590e3f2e492-data-volume\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.473577 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.473548 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/305b0ad5-11e3-4960-9666-7590e3f2e492-crio-socket\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.492340 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.492306 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkm5v\" (UniqueName: \"kubernetes.io/projected/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-api-access-wkm5v\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.535024 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.534954 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:49.535024 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.535019 2874 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:57:49.978382 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.978346 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.978580 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:49.978483 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:49.978580 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:49.978524 2874 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:57:49.978674 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:49.978603 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls podName:305b0ad5-11e3-4960-9666-7590e3f2e492 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:50.978583405 +0000 UTC m=+244.312948098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gg9g5" (UID: "305b0ad5-11e3-4960-9666-7590e3f2e492") : secret "insights-runtime-extractor-tls" not found Sep 18 16:57:49.978674 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:49.978619 2874 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:57:49.978762 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:49.978688 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm podName:305b0ad5-11e3-4960-9666-7590e3f2e492 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:50.978675267 +0000 UTC m=+244.313039956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gg9g5" (UID: "305b0ad5-11e3-4960-9666-7590e3f2e492") : configmap "kube-rbac-proxy" not found Sep 18 16:57:50.243708 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:50.243614 2874 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:57:50.244591 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:50.244562 2874 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:57:50.245442 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:50.245421 2874 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:57:50.245510 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:50.245452 2874 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" podUID="1e131f4d-c737-4144-967f-cebe6ed736c1" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 18 16:57:50.300718 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:50.300695 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 16:57:50.301051 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:50.301033 2874 scope.go:117] "RemoveContainer" containerID="5a1200963adf617cc38f81698af838f063fbbcaff92da93f784625037efbe1fd" Sep 18 16:57:50.301186 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:50.301170 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-v5qkh_openshift-console-operator(0d14e37c-3b2e-4192-8e48-8cc1bfba3707)\"" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" podUID="0d14e37c-3b2e-4192-8e48-8cc1bfba3707" Sep 18 16:57:50.813061 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:50.813032 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l6cpb_175a17f1-cd97-40d2-b70a-e8ba235c28eb/dns-node-resolver/0.log" Sep 18 16:57:50.988527 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:50.988491 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:50.988690 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:50.988604 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:50.988690 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:50.988616 2874 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:57:50.988690 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:50.988675 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm podName:305b0ad5-11e3-4960-9666-7590e3f2e492 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:52.988659491 +0000 UTC m=+246.323024192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gg9g5" (UID: "305b0ad5-11e3-4960-9666-7590e3f2e492") : configmap "kube-rbac-proxy" not found Sep 18 16:57:50.988799 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:50.988703 2874 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:57:50.988799 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:50.988742 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls podName:305b0ad5-11e3-4960-9666-7590e3f2e492 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:52.98873167 +0000 UTC m=+246.323096369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gg9g5" (UID: "305b0ad5-11e3-4960-9666-7590e3f2e492") : secret "insights-runtime-extractor-tls" not found Sep 18 16:57:51.304598 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:51.304569 2874 scope.go:117] "RemoveContainer" containerID="5a1200963adf617cc38f81698af838f063fbbcaff92da93f784625037efbe1fd" Sep 18 16:57:51.304969 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:51.304728 2874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-v5qkh_openshift-console-operator(0d14e37c-3b2e-4192-8e48-8cc1bfba3707)\"" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" podUID="0d14e37c-3b2e-4192-8e48-8cc1bfba3707" Sep 18 16:57:51.612510 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:51.612432 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-br879_eb9022da-0935-4c13-9fdc-102da03b843e/node-ca/0.log" Sep 18 16:57:52.412947 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:52.412905 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bdv7m_8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0/serve-healthcheck-canary/0.log" Sep 18 16:57:53.009821 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:53.009785 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:53.010038 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:53.009864 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:53.010038 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:53.009962 2874 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:57:53.010038 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:53.009964 2874 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:57:53.010038 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:53.010014 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm podName:305b0ad5-11e3-4960-9666-7590e3f2e492 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:57.010001311 +0000 UTC m=+250.344366000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gg9g5" (UID: "305b0ad5-11e3-4960-9666-7590e3f2e492") : configmap "kube-rbac-proxy" not found Sep 18 16:57:53.010038 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:53.010029 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls podName:305b0ad5-11e3-4960-9666-7590e3f2e492 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:57.010021772 +0000 UTC m=+250.344386457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gg9g5" (UID: "305b0ad5-11e3-4960-9666-7590e3f2e492") : secret "insights-runtime-extractor-tls" not found Sep 18 16:57:55.029342 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:55.029303 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:55.029746 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:55.029422 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:57:55.029746 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:55.029459 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:57:55.029746 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.029471 2874 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 16:57:55.029746 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.029536 2874 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 16:57:55.029746 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.029550 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:58:11.02953406 +0000 UTC m=+264.363898748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : secret "router-metrics-certs-default" not found Sep 18 16:57:55.029746 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.029571 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle podName:7897222e-78f1-4920-9afe-669e89dffcab nodeName:}" failed. No retries permitted until 2026-09-18 16:58:11.029557641 +0000 UTC m=+264.363922330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle") pod "router-default-7ccfc6f4f8-ksblq" (UID: "7897222e-78f1-4920-9afe-669e89dffcab") : configmap references non-existent config key: service-ca.crt Sep 18 16:57:55.029746 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.029592 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls podName:0c2d3679-f1c1-4769-9266-bcee43b95806 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:11.029585591 +0000 UTC m=+264.363950279 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b5ljd" (UID: "0c2d3679-f1c1-4769-9266-bcee43b95806") : secret "samples-operator-tls" not found Sep 18 16:57:55.130695 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:55.130653 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:57:55.130890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:55.130716 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:57:55.130890 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.130789 2874 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:55.130890 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.130801 2874 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 18 16:57:55.130890 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:55.130831 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:57:55.130890 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.130847 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls podName:a9d00046-58ab-4f9c-b9bd-85b4853f2fd8 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:11.130834192 +0000 UTC m=+264.465198876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7gbc6" (UID: "a9d00046-58ab-4f9c-b9bd-85b4853f2fd8") : secret "cluster-monitoring-operator-tls" not found Sep 18 16:57:55.131093 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.130892 2874 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:57:55.131093 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.130904 2874 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b5b5c5f4-tpsp6: secret "image-registry-tls" not found Sep 18 16:57:55.131093 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.130920 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert podName:1de7a1b3-6bc5-4f4c-832a-677bd92764f1 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:11.130901239 +0000 UTC m=+264.465265928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-w2jnm" (UID: "1de7a1b3-6bc5-4f4c-832a-677bd92764f1") : secret "networking-console-plugin-cert" not found Sep 18 16:57:55.131093 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.130945 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls podName:5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:11.130935881 +0000 UTC m=+264.465300567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls") pod "image-registry-56b5b5c5f4-tpsp6" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1") : secret "image-registry-tls" not found Sep 18 16:57:55.332812 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:55.332720 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:57:55.333068 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.332897 2874 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:57:55.333068 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:55.332973 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls podName:3fab1373-7e52-4716-ae69-44a8d87cc7ae nodeName:}" failed. No retries permitted until 2026-09-18 16:58:11.33295379 +0000 UTC m=+264.667318477 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls") pod "dns-default-gklt9" (UID: "3fab1373-7e52-4716-ae69-44a8d87cc7ae") : secret "dns-default-metrics-tls" not found Sep 18 16:57:57.047585 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:57.047549 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:57.048008 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:57:57.047634 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:57:57.048008 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:57.047711 2874 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:57:57.048008 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:57.047726 2874 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:57:57.048008 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:57.047776 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm podName:305b0ad5-11e3-4960-9666-7590e3f2e492 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:05.047762777 +0000 UTC m=+258.382127461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gg9g5" (UID: "305b0ad5-11e3-4960-9666-7590e3f2e492") : configmap "kube-rbac-proxy" not found Sep 18 16:57:57.048008 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:57:57.047788 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls podName:305b0ad5-11e3-4960-9666-7590e3f2e492 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:05.047782876 +0000 UTC m=+258.382147561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gg9g5" (UID: "305b0ad5-11e3-4960-9666-7590e3f2e492") : secret "insights-runtime-extractor-tls" not found Sep 18 16:58:00.243200 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:00.243158 2874 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:58:00.244106 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:00.244079 2874 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:58:00.244993 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:00.244960 2874 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:58:00.245074 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:00.245004 2874 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" podUID="1e131f4d-c737-4144-967f-cebe6ed736c1" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 18 16:58:02.803613 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:02.803566 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:58:02.804056 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:02.803651 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:58:02.806302 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:02.806282 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 16:58:02.807268 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:02.807252 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 16:58:02.816608 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:02.816584 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb0c72b0-0d55-4136-99ce-d50036c55d91-metrics-certs\") pod \"network-metrics-daemon-mqrhp\" (UID: \"bb0c72b0-0d55-4136-99ce-d50036c55d91\") " pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:58:02.816608 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:02.816598 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb442e64-9a04-459c-814a-71a5bf49c5f7-original-pull-secret\") pod \"global-pull-secret-syncer-t4fzq\" (UID: \"cb442e64-9a04-459c-814a-71a5bf49c5f7\") " pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:58:02.854018 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:02.853990 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4fzq" Sep 18 16:58:02.862143 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:02.862120 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d4kk4\"" Sep 18 16:58:02.869776 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:02.869746 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mqrhp" Sep 18 16:58:03.122748 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:03.122553 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t4fzq"] Sep 18 16:58:03.125449 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:03.125420 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb442e64_9a04_459c_814a_71a5bf49c5f7.slice/crio-f3833853794def53a10ab0ddd4d1cf5bd87a403ca21372b404c05595e875bb7c WatchSource:0}: Error finding container f3833853794def53a10ab0ddd4d1cf5bd87a403ca21372b404c05595e875bb7c: Status 404 returned error can't find the container with id f3833853794def53a10ab0ddd4d1cf5bd87a403ca21372b404c05595e875bb7c Sep 18 16:58:03.147897 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:03.147874 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mqrhp"] Sep 18 16:58:03.150203 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:03.150171 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0c72b0_0d55_4136_99ce_d50036c55d91.slice/crio-f5a0e8865e0fbd77d86e64c71bb06bc1f98f1983977449a41f2c9941500d3170 WatchSource:0}: Error finding container f5a0e8865e0fbd77d86e64c71bb06bc1f98f1983977449a41f2c9941500d3170: Status 404 returned error can't find the container with id f5a0e8865e0fbd77d86e64c71bb06bc1f98f1983977449a41f2c9941500d3170 Sep 18 16:58:03.348272 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:03.348232 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mqrhp" event={"ID":"bb0c72b0-0d55-4136-99ce-d50036c55d91","Type":"ContainerStarted","Data":"f5a0e8865e0fbd77d86e64c71bb06bc1f98f1983977449a41f2c9941500d3170"} Sep 18 16:58:03.349266 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:03.349245 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t4fzq" event={"ID":"cb442e64-9a04-459c-814a-71a5bf49c5f7","Type":"ContainerStarted","Data":"f3833853794def53a10ab0ddd4d1cf5bd87a403ca21372b404c05595e875bb7c"} Sep 18 16:58:04.236044 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:04.236019 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsvlw" Sep 18 16:58:04.354854 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:04.354820 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mqrhp" event={"ID":"bb0c72b0-0d55-4136-99ce-d50036c55d91","Type":"ContainerStarted","Data":"da332bb8344812602cf1474da705e7d27b9bd826e29d41f6b7fd5f61e347adf4"} Sep 18 16:58:05.125495 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.125445 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:58:05.125682 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.125555 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:58:05.126108 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.126078 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/305b0ad5-11e3-4960-9666-7590e3f2e492-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:58:05.128129 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.128091 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/305b0ad5-11e3-4960-9666-7590e3f2e492-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gg9g5\" (UID: \"305b0ad5-11e3-4960-9666-7590e3f2e492\") " pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:58:05.200598 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.200560 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gg9g5" Sep 18 16:58:05.366390 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.366298 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mqrhp" event={"ID":"bb0c72b0-0d55-4136-99ce-d50036c55d91","Type":"ContainerStarted","Data":"1294b80eefd79f71c72b2db6614a089bf66ae80ea9c4c2007337e047a593d198"} Sep 18 16:58:05.383415 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.383365 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mqrhp" podStartSLOduration=69.380608874 podStartE2EDuration="1m10.383350175s" totalImagesPullingTime="1.002741301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:05.383009428 +0000 UTC m=+258.717374136" watchObservedRunningTime="2026-09-18 16:58:05.383350175 +0000 UTC m=+258.717714882" Sep 18 16:58:05.428551 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.428509 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dcrn\" (UniqueName: \"kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn\") pod \"network-check-target-npjd7\" (UID: \"e9379c30-e23f-4635-a9c1-73d866893686\") " pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:58:05.431810 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.431783 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dcrn\" (UniqueName: \"kubernetes.io/projected/e9379c30-e23f-4635-a9c1-73d866893686-kube-api-access-9dcrn\") pod \"network-check-target-npjd7\" (UID: \"e9379c30-e23f-4635-a9c1-73d866893686\") " pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:58:05.481690 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.481666 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gg9g5"] Sep 18 16:58:05.484551 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:05.484526 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305b0ad5_11e3_4960_9666_7590e3f2e492.slice/crio-2dc9ed4a6df0f3e1b0d8c602b60f404237c8d1113bea5ccb3b1121d5ab4f2975 WatchSource:0}: Error finding container 2dc9ed4a6df0f3e1b0d8c602b60f404237c8d1113bea5ccb3b1121d5ab4f2975: Status 404 returned error can't find the container with id 2dc9ed4a6df0f3e1b0d8c602b60f404237c8d1113bea5ccb3b1121d5ab4f2975 Sep 18 16:58:05.566533 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.566503 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-plhlv\"" Sep 18 16:58:05.573969 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.573944 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:58:05.743293 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.743214 2874 scope.go:117] "RemoveContainer" containerID="5a1200963adf617cc38f81698af838f063fbbcaff92da93f784625037efbe1fd" Sep 18 16:58:05.849297 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:05.849265 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-npjd7"] Sep 18 16:58:06.372916 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:06.372873 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gg9g5" event={"ID":"305b0ad5-11e3-4960-9666-7590e3f2e492","Type":"ContainerStarted","Data":"47539a34913740d60ff693a1d2a4af5fd3a7db4c145ea091e12b2ebcb20e429b"} Sep 18 16:58:06.373428 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:06.372953 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gg9g5" event={"ID":"305b0ad5-11e3-4960-9666-7590e3f2e492","Type":"ContainerStarted","Data":"2dc9ed4a6df0f3e1b0d8c602b60f404237c8d1113bea5ccb3b1121d5ab4f2975"} Sep 18 16:58:06.994392 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:06.994180 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9379c30_e23f_4635_a9c1_73d866893686.slice/crio-f036ab468e9b28b93770bc9f7e2327675ddcaa12e9192f70510c2ca413411fbe WatchSource:0}: Error finding container f036ab468e9b28b93770bc9f7e2327675ddcaa12e9192f70510c2ca413411fbe: Status 404 returned error can't find the container with id f036ab468e9b28b93770bc9f7e2327675ddcaa12e9192f70510c2ca413411fbe Sep 18 16:58:07.380668 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.380640 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 16:58:07.381135 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.380751 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" event={"ID":"0d14e37c-3b2e-4192-8e48-8cc1bfba3707","Type":"ContainerStarted","Data":"fdae5ef43afc96884f8c1b02187b06e6f4f5918d3cce2c4f3a2d514283bf5c84"} Sep 18 16:58:07.381135 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.381057 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:58:07.382388 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.382358 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-npjd7" event={"ID":"e9379c30-e23f-4635-a9c1-73d866893686","Type":"ContainerStarted","Data":"fc0848c4dbcee8285a7556b2a594938f5b20ff0a98160f460e52c050393abfc8"} Sep 18 16:58:07.382556 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.382434 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-npjd7" event={"ID":"e9379c30-e23f-4635-a9c1-73d866893686","Type":"ContainerStarted","Data":"f036ab468e9b28b93770bc9f7e2327675ddcaa12e9192f70510c2ca413411fbe"} Sep 18 16:58:07.382556 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.382466 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:58:07.383985 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.383956 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t4fzq" event={"ID":"cb442e64-9a04-459c-814a-71a5bf49c5f7","Type":"ContainerStarted","Data":"0529d6fadb29d5e73731c1e732501901a21106abeb43f6bdac3ca91c49fc48ac"} Sep 18 16:58:07.385857 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.385838 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gg9g5" event={"ID":"305b0ad5-11e3-4960-9666-7590e3f2e492","Type":"ContainerStarted","Data":"61fa04f5f5b95dd8a61518109eecf00a94c4d9cad55b8aec966c4f6face97fe6"} Sep 18 16:58:07.396945 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.396898 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" podStartSLOduration=64.927639208 podStartE2EDuration="1m12.396879131s" totalImagesPullingTime="7.469239923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:07.395676736 +0000 UTC m=+260.730041443" watchObservedRunningTime="2026-09-18 16:58:07.396879131 +0000 UTC m=+260.731243839" Sep 18 16:58:07.412197 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.412143 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-npjd7" podStartSLOduration=66.412129184 podStartE2EDuration="1m6.412129184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:07.411274062 +0000 UTC m=+260.745638768" watchObservedRunningTime="2026-09-18 16:58:07.412129184 +0000 UTC m=+260.746493891" Sep 18 16:58:07.616654 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.616563 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-v5qkh" Sep 18 16:58:07.633256 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:07.633194 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t4fzq" podStartSLOduration=68.726272186 podStartE2EDuration="1m12.63317443s" totalImagesPullingTime="3.906902244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:07.426940696 +0000 UTC m=+260.761305403" watchObservedRunningTime="2026-09-18 16:58:07.63317443 +0000 UTC m=+260.967539137" Sep 18 16:58:09.394893 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:09.394855 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gg9g5" event={"ID":"305b0ad5-11e3-4960-9666-7590e3f2e492","Type":"ContainerStarted","Data":"d1640ea38c14513055d13c0591e90f19cc4aae6e690f9c136c839a5b8fb33a31"} Sep 18 16:58:09.411772 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:09.411720 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gg9g5" podStartSLOduration=17.548767091 podStartE2EDuration="20.411704029s" totalImagesPullingTime="2.862936938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:09.4103841 +0000 UTC m=+262.744748807" watchObservedRunningTime="2026-09-18 16:58:09.411704029 +0000 UTC m=+262.746068735" Sep 18 16:58:10.243512 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:10.243471 2874 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:58:10.244550 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:10.244516 2874 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:58:10.245419 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:10.245375 2874 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:58:10.245502 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:10.245433 2874 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" podUID="1e131f4d-c737-4144-967f-cebe6ed736c1" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 18 16:58:11.077802 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.077769 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:58:11.078192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.077844 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:58:11.078192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.077880 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:58:11.078485 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.078465 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7897222e-78f1-4920-9afe-669e89dffcab-service-ca-bundle\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:58:11.080259 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.080231 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7897222e-78f1-4920-9afe-669e89dffcab-metrics-certs\") pod \"router-default-7ccfc6f4f8-ksblq\" (UID: \"7897222e-78f1-4920-9afe-669e89dffcab\") " pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:58:11.080259 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.080249 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c2d3679-f1c1-4769-9266-bcee43b95806-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b5ljd\" (UID: \"0c2d3679-f1c1-4769-9266-bcee43b95806\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:58:11.178515 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.178481 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:58:11.178678 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.178528 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:58:11.178678 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.178655 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:58:11.181011 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.180989 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") pod \"image-registry-56b5b5c5f4-tpsp6\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:58:11.181168 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.181147 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9d00046-58ab-4f9c-b9bd-85b4853f2fd8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7gbc6\" (UID: \"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:58:11.181244 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.181147 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1de7a1b3-6bc5-4f4c-832a-677bd92764f1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-w2jnm\" (UID: \"1de7a1b3-6bc5-4f4c-832a-677bd92764f1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:58:11.344453 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.344356 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dtp4g\"" Sep 18 16:58:11.351681 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.351655 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" Sep 18 16:58:11.351906 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.351889 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9wmn4\"" Sep 18 16:58:11.360796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.360676 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:58:11.380931 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.380900 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:58:11.383979 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.383538 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fab1373-7e52-4716-ae69-44a8d87cc7ae-metrics-tls\") pod \"dns-default-gklt9\" (UID: \"3fab1373-7e52-4716-ae69-44a8d87cc7ae\") " pod="openshift-dns/dns-default-gklt9" Sep 18 16:58:11.386681 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.386661 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-djfp2\"" Sep 18 16:58:11.393919 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.393896 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" Sep 18 16:58:11.431358 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.431332 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fmtn2\"" Sep 18 16:58:11.439143 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.438917 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:58:11.441619 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.441588 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-m5qs4\"" Sep 18 16:58:11.450221 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.449978 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" Sep 18 16:58:11.566928 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.566906 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kgcdr\"" Sep 18 16:58:11.574984 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.574961 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gklt9" Sep 18 16:58:11.644810 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.644664 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd"] Sep 18 16:58:11.645769 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.645751 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7ccfc6f4f8-ksblq"] Sep 18 16:58:11.647135 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:11.647108 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7897222e_78f1_4920_9afe_669e89dffcab.slice/crio-6394ccd9756a9072387b1e6fd2d85a1c51bfa60b66837341da994be75937d454 WatchSource:0}: Error finding container 6394ccd9756a9072387b1e6fd2d85a1c51bfa60b66837341da994be75937d454: Status 404 returned error can't find the container with id 6394ccd9756a9072387b1e6fd2d85a1c51bfa60b66837341da994be75937d454 Sep 18 16:58:11.681181 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.680907 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6"] Sep 18 16:58:11.683026 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:11.683000 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d00046_58ab_4f9c_b9bd_85b4853f2fd8.slice/crio-11221f78712d8eb62e165954773131b42de37651eb985970dbe1136b1dcea068 WatchSource:0}: Error finding container 11221f78712d8eb62e165954773131b42de37651eb985970dbe1136b1dcea068: Status 404 returned error can't find the container with id 11221f78712d8eb62e165954773131b42de37651eb985970dbe1136b1dcea068 Sep 18 16:58:11.733673 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.733589 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56b5b5c5f4-tpsp6"] Sep 18 16:58:11.736560 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:11.736535 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebffbf1_54a9_4ec9_9e8e_130aeca4e7c1.slice/crio-6867561ada2cd1ad934824106fb4b8a15b4c1bb8f006111833e211d34b124c28 WatchSource:0}: Error finding container 6867561ada2cd1ad934824106fb4b8a15b4c1bb8f006111833e211d34b124c28: Status 404 returned error can't find the container with id 6867561ada2cd1ad934824106fb4b8a15b4c1bb8f006111833e211d34b124c28 Sep 18 16:58:11.754270 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.754241 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm"] Sep 18 16:58:11.766192 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:11.766163 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de7a1b3_6bc5_4f4c_832a_677bd92764f1.slice/crio-fa9d1a7ee5de9f12390852d128362bdb35c183583f5ec34df40492cd2afd71b9 WatchSource:0}: Error finding container fa9d1a7ee5de9f12390852d128362bdb35c183583f5ec34df40492cd2afd71b9: Status 404 returned error can't find the container with id fa9d1a7ee5de9f12390852d128362bdb35c183583f5ec34df40492cd2afd71b9 Sep 18 16:58:11.847889 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:11.847855 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gklt9"] Sep 18 16:58:11.850695 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:11.850671 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fab1373_7e52_4716_ae69_44a8d87cc7ae.slice/crio-dffed31b5948211577af2b33c38df660ffbef51725b58263128978ffe8537753 WatchSource:0}: Error finding container dffed31b5948211577af2b33c38df660ffbef51725b58263128978ffe8537753: Status 404 returned error can't find the container with id dffed31b5948211577af2b33c38df660ffbef51725b58263128978ffe8537753 Sep 18 16:58:12.410458 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.410312 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" event={"ID":"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8","Type":"ContainerStarted","Data":"11221f78712d8eb62e165954773131b42de37651eb985970dbe1136b1dcea068"} Sep 18 16:58:12.413560 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.413022 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" event={"ID":"7897222e-78f1-4920-9afe-669e89dffcab","Type":"ContainerStarted","Data":"72f17317a1eca12acd182a89525ecbdb2792f79b75d89e51225e95b44d927ec7"} Sep 18 16:58:12.413560 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.413073 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" event={"ID":"7897222e-78f1-4920-9afe-669e89dffcab","Type":"ContainerStarted","Data":"6394ccd9756a9072387b1e6fd2d85a1c51bfa60b66837341da994be75937d454"} Sep 18 16:58:12.414825 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.414794 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gklt9" event={"ID":"3fab1373-7e52-4716-ae69-44a8d87cc7ae","Type":"ContainerStarted","Data":"dffed31b5948211577af2b33c38df660ffbef51725b58263128978ffe8537753"} Sep 18 16:58:12.416802 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.416766 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" event={"ID":"1de7a1b3-6bc5-4f4c-832a-677bd92764f1","Type":"ContainerStarted","Data":"fa9d1a7ee5de9f12390852d128362bdb35c183583f5ec34df40492cd2afd71b9"} Sep 18 16:58:12.418999 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.418971 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" event={"ID":"0c2d3679-f1c1-4769-9266-bcee43b95806","Type":"ContainerStarted","Data":"8a8f5e07f4e70cc6297c89747889e2b0f160f99bfd9426039bfbcd2f3b562ec0"} Sep 18 16:58:12.421377 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.420757 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" event={"ID":"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1","Type":"ContainerStarted","Data":"e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be"} Sep 18 16:58:12.421377 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.420798 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" event={"ID":"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1","Type":"ContainerStarted","Data":"6867561ada2cd1ad934824106fb4b8a15b4c1bb8f006111833e211d34b124c28"} Sep 18 16:58:12.421377 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.421338 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:58:12.434191 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.433989 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" podStartSLOduration=77.433970339 podStartE2EDuration="1m17.433970339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:12.431251952 +0000 UTC m=+265.765616685" watchObservedRunningTime="2026-09-18 16:58:12.433970339 +0000 UTC m=+265.768335049" Sep 18 16:58:12.452335 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:12.452280 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" podStartSLOduration=69.452265353 podStartE2EDuration="1m9.452265353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:12.451997845 +0000 UTC m=+265.786362553" watchObservedRunningTime="2026-09-18 16:58:12.452265353 +0000 UTC m=+265.786630060" Sep 18 16:58:13.361481 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:13.361450 2874 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:58:13.364624 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:13.364599 2874 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:58:13.425274 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:13.425241 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:58:13.426681 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:13.426659 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7ccfc6f4f8-ksblq" Sep 18 16:58:15.434144 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.434117 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-qqdzb_1e131f4d-c737-4144-967f-cebe6ed736c1/kube-multus-additional-cni-plugins/0.log" Sep 18 16:58:15.434511 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.434153 2874 generic.go:356] "Generic (PLEG): container finished" podID="1e131f4d-c737-4144-967f-cebe6ed736c1" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" exitCode=137 Sep 18 16:58:15.434511 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.434232 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" event={"ID":"1e131f4d-c737-4144-967f-cebe6ed736c1","Type":"ContainerDied","Data":"4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea"} Sep 18 16:58:15.546619 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.546595 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-qqdzb_1e131f4d-c737-4144-967f-cebe6ed736c1/kube-multus-additional-cni-plugins/0.log" Sep 18 16:58:15.567850 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.567830 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:58:15.617773 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.617740 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sb78\" (UniqueName: \"kubernetes.io/projected/1e131f4d-c737-4144-967f-cebe6ed736c1-kube-api-access-5sb78\") pod \"1e131f4d-c737-4144-967f-cebe6ed736c1\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " Sep 18 16:58:15.617884 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.617802 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e131f4d-c737-4144-967f-cebe6ed736c1-tuning-conf-dir\") pod \"1e131f4d-c737-4144-967f-cebe6ed736c1\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " Sep 18 16:58:15.617884 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.617829 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e131f4d-c737-4144-967f-cebe6ed736c1-cni-sysctl-allowlist\") pod \"1e131f4d-c737-4144-967f-cebe6ed736c1\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " Sep 18 16:58:15.618004 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.617879 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/1e131f4d-c737-4144-967f-cebe6ed736c1-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "1e131f4d-c737-4144-967f-cebe6ed736c1" (UID: "1e131f4d-c737-4144-967f-cebe6ed736c1"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:58:15.618004 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.617893 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/1e131f4d-c737-4144-967f-cebe6ed736c1-ready\") pod \"1e131f4d-c737-4144-967f-cebe6ed736c1\" (UID: \"1e131f4d-c737-4144-967f-cebe6ed736c1\") " Sep 18 16:58:15.618077 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.618057 2874 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e131f4d-c737-4144-967f-cebe6ed736c1-tuning-conf-dir\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:15.618201 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.618181 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e131f4d-c737-4144-967f-cebe6ed736c1-ready" (OuterVolumeSpecName: "ready") pod "1e131f4d-c737-4144-967f-cebe6ed736c1" (UID: "1e131f4d-c737-4144-967f-cebe6ed736c1"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:58:15.618280 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.618198 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e131f4d-c737-4144-967f-cebe6ed736c1-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "1e131f4d-c737-4144-967f-cebe6ed736c1" (UID: "1e131f4d-c737-4144-967f-cebe6ed736c1"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:58:15.620258 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.620237 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e131f4d-c737-4144-967f-cebe6ed736c1-kube-api-access-5sb78" (OuterVolumeSpecName: "kube-api-access-5sb78") pod "1e131f4d-c737-4144-967f-cebe6ed736c1" (UID: "1e131f4d-c737-4144-967f-cebe6ed736c1"). InnerVolumeSpecName "kube-api-access-5sb78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:58:15.678143 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.678117 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp"] Sep 18 16:58:15.678376 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.678365 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e131f4d-c737-4144-967f-cebe6ed736c1" containerName="kube-multus-additional-cni-plugins" Sep 18 16:58:15.678492 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.678378 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e131f4d-c737-4144-967f-cebe6ed736c1" containerName="kube-multus-additional-cni-plugins" Sep 18 16:58:15.678782 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.678758 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e131f4d-c737-4144-967f-cebe6ed736c1" containerName="kube-multus-additional-cni-plugins" Sep 18 16:58:15.702803 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.702776 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp"] Sep 18 16:58:15.702902 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.702863 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:15.705883 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.705850 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 18 16:58:15.705994 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.705960 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 18 16:58:15.706053 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.706016 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 18 16:58:15.706112 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.706098 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 18 16:58:15.719089 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.719070 2874 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/1e131f4d-c737-4144-967f-cebe6ed736c1-ready\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:15.719178 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.719091 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sb78\" (UniqueName: \"kubernetes.io/projected/1e131f4d-c737-4144-967f-cebe6ed736c1-kube-api-access-5sb78\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:15.719178 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.719101 2874 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e131f4d-c737-4144-967f-cebe6ed736c1-cni-sysctl-allowlist\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:15.819598 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.819454 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkklh\" (UniqueName: \"kubernetes.io/projected/6dcbcdc3-43bf-4684-b600-7e967372b6ca-kube-api-access-kkklh\") pod \"klusterlet-addon-workmgr-ffd896b95-vj7lp\" (UID: \"6dcbcdc3-43bf-4684-b600-7e967372b6ca\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:15.819598 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.819531 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dcbcdc3-43bf-4684-b600-7e967372b6ca-tmp\") pod \"klusterlet-addon-workmgr-ffd896b95-vj7lp\" (UID: \"6dcbcdc3-43bf-4684-b600-7e967372b6ca\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:15.819796 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.819660 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6dcbcdc3-43bf-4684-b600-7e967372b6ca-klusterlet-config\") pod \"klusterlet-addon-workmgr-ffd896b95-vj7lp\" (UID: \"6dcbcdc3-43bf-4684-b600-7e967372b6ca\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:15.920249 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.920203 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6dcbcdc3-43bf-4684-b600-7e967372b6ca-klusterlet-config\") pod \"klusterlet-addon-workmgr-ffd896b95-vj7lp\" (UID: \"6dcbcdc3-43bf-4684-b600-7e967372b6ca\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:15.920387 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.920264 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkklh\" (UniqueName: \"kubernetes.io/projected/6dcbcdc3-43bf-4684-b600-7e967372b6ca-kube-api-access-kkklh\") pod \"klusterlet-addon-workmgr-ffd896b95-vj7lp\" (UID: \"6dcbcdc3-43bf-4684-b600-7e967372b6ca\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:15.920387 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.920304 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dcbcdc3-43bf-4684-b600-7e967372b6ca-tmp\") pod \"klusterlet-addon-workmgr-ffd896b95-vj7lp\" (UID: \"6dcbcdc3-43bf-4684-b600-7e967372b6ca\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:15.920852 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.920805 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dcbcdc3-43bf-4684-b600-7e967372b6ca-tmp\") pod \"klusterlet-addon-workmgr-ffd896b95-vj7lp\" (UID: \"6dcbcdc3-43bf-4684-b600-7e967372b6ca\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:15.923085 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.923065 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6dcbcdc3-43bf-4684-b600-7e967372b6ca-klusterlet-config\") pod \"klusterlet-addon-workmgr-ffd896b95-vj7lp\" (UID: \"6dcbcdc3-43bf-4684-b600-7e967372b6ca\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:15.929993 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:15.929959 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkklh\" (UniqueName: \"kubernetes.io/projected/6dcbcdc3-43bf-4684-b600-7e967372b6ca-kube-api-access-kkklh\") pod \"klusterlet-addon-workmgr-ffd896b95-vj7lp\" (UID: \"6dcbcdc3-43bf-4684-b600-7e967372b6ca\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:16.015364 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.015331 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:16.291962 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.291934 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf"] Sep 18 16:58:16.314641 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:16.314612 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcbcdc3_43bf_4684_b600_7e967372b6ca.slice/crio-3558ccf4d69706c6b9585b64d2623601bfa7ce74d42efb0ad1e68935df2c3ca2 WatchSource:0}: Error finding container 3558ccf4d69706c6b9585b64d2623601bfa7ce74d42efb0ad1e68935df2c3ca2: Status 404 returned error can't find the container with id 3558ccf4d69706c6b9585b64d2623601bfa7ce74d42efb0ad1e68935df2c3ca2 Sep 18 16:58:16.318998 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.318950 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf"] Sep 18 16:58:16.318998 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.319000 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp"] Sep 18 16:58:16.319152 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.319065 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" Sep 18 16:58:16.321916 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.321883 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 18 16:58:16.322017 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.321978 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-x8c6b\"" Sep 18 16:58:16.425841 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.425802 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2a3737fa-5bf1-4d61-a683-7028764b81e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-jckmf\" (UID: \"2a3737fa-5bf1-4d61-a683-7028764b81e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" Sep 18 16:58:16.439608 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.439572 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" event={"ID":"a9d00046-58ab-4f9c-b9bd-85b4853f2fd8","Type":"ContainerStarted","Data":"877ac01ff0910c9109e65ada810c8a78d6921b4945f358eccd9b82f332ef44f2"} Sep 18 16:58:16.441529 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.441505 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gklt9" event={"ID":"3fab1373-7e52-4716-ae69-44a8d87cc7ae","Type":"ContainerStarted","Data":"7db541978e66dbe7b45fa101677c4a634973843aecd26941bf88728687d59759"} Sep 18 16:58:16.441671 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.441561 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gklt9" event={"ID":"3fab1373-7e52-4716-ae69-44a8d87cc7ae","Type":"ContainerStarted","Data":"f0a5e454ff6eb13f3f606e006769b552ae3a6f1268aa3efc423fa33d7d5cbe1c"} Sep 18 16:58:16.441671 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.441593 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gklt9" Sep 18 16:58:16.443065 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.443045 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" event={"ID":"1de7a1b3-6bc5-4f4c-832a-677bd92764f1","Type":"ContainerStarted","Data":"3796872205b5f68bb659e3e80f5205ab4a1da5963fca166f2b5f1072d74b2c38"} Sep 18 16:58:16.444866 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.444846 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" event={"ID":"0c2d3679-f1c1-4769-9266-bcee43b95806","Type":"ContainerStarted","Data":"d90d8b6d03c4ae449dc04fb0180407a5155732cc8d5f55752bad890adeb7af03"} Sep 18 16:58:16.444980 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.444893 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" event={"ID":"0c2d3679-f1c1-4769-9266-bcee43b95806","Type":"ContainerStarted","Data":"2c443c79859f0cd90148b53630a719093e7514196c758d4ac5ba1de5f4568211"} Sep 18 16:58:16.446057 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.446035 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" event={"ID":"6dcbcdc3-43bf-4684-b600-7e967372b6ca","Type":"ContainerStarted","Data":"3558ccf4d69706c6b9585b64d2623601bfa7ce74d42efb0ad1e68935df2c3ca2"} Sep 18 16:58:16.447274 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.447257 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-qqdzb_1e131f4d-c737-4144-967f-cebe6ed736c1/kube-multus-additional-cni-plugins/0.log" Sep 18 16:58:16.447352 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.447308 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" event={"ID":"1e131f4d-c737-4144-967f-cebe6ed736c1","Type":"ContainerDied","Data":"bc74940a714ff1faba43535f8dc7c9acf53c6c5f3e6f46ac218abb4fefa52fb4"} Sep 18 16:58:16.447352 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.447340 2874 scope.go:117] "RemoveContainer" containerID="4ef3f6ce53d5086682c7a724d8c6b8bae4e69668ee573c665196c5e2246a4fea" Sep 18 16:58:16.447352 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.447346 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-qqdzb" Sep 18 16:58:16.457385 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.457335 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7gbc6" podStartSLOduration=77.49086447 podStartE2EDuration="1m21.45731689s" totalImagesPullingTime="3.96645242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:16.455982591 +0000 UTC m=+269.790347299" watchObservedRunningTime="2026-09-18 16:58:16.45731689 +0000 UTC m=+269.791681599" Sep 18 16:58:16.470758 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.470708 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-w2jnm" podStartSLOduration=67.592369062 podStartE2EDuration="1m11.470692592s" totalImagesPullingTime="3.87832353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:16.469344823 +0000 UTC m=+269.803709531" watchObservedRunningTime="2026-09-18 16:58:16.470692592 +0000 UTC m=+269.805057299" Sep 18 16:58:16.491707 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.491666 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gklt9" podStartSLOduration=33.697977618 podStartE2EDuration="37.491654167s" totalImagesPullingTime="3.793676549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:16.490902512 +0000 UTC m=+269.825267211" watchObservedRunningTime="2026-09-18 16:58:16.491654167 +0000 UTC m=+269.826018874" Sep 18 16:58:16.509031 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.508988 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b5ljd" podStartSLOduration=77.558819677 podStartE2EDuration="1m21.508971206s" totalImagesPullingTime="3.950151529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:16.507139015 +0000 UTC m=+269.841503724" watchObservedRunningTime="2026-09-18 16:58:16.508971206 +0000 UTC m=+269.843335914" Sep 18 16:58:16.526735 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.526709 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2a3737fa-5bf1-4d61-a683-7028764b81e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-jckmf\" (UID: \"2a3737fa-5bf1-4d61-a683-7028764b81e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" Sep 18 16:58:16.527449 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:16.527055 2874 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 18 16:58:16.527449 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:16.527122 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a3737fa-5bf1-4d61-a683-7028764b81e3-tls-certificates podName:2a3737fa-5bf1-4d61-a683-7028764b81e3 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:17.027104374 +0000 UTC m=+270.361469063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/2a3737fa-5bf1-4d61-a683-7028764b81e3-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-jckmf" (UID: "2a3737fa-5bf1-4d61-a683-7028764b81e3") : secret "prometheus-operator-admission-webhook-tls" not found Sep 18 16:58:16.529465 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.529435 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-qqdzb"] Sep 18 16:58:16.532907 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.532885 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-qqdzb"] Sep 18 16:58:16.748179 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:16.748144 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e131f4d-c737-4144-967f-cebe6ed736c1" path="/var/lib/kubelet/pods/1e131f4d-c737-4144-967f-cebe6ed736c1/volumes" Sep 18 16:58:17.031132 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:17.031041 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2a3737fa-5bf1-4d61-a683-7028764b81e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-jckmf\" (UID: \"2a3737fa-5bf1-4d61-a683-7028764b81e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" Sep 18 16:58:17.033671 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:17.033649 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2a3737fa-5bf1-4d61-a683-7028764b81e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-jckmf\" (UID: \"2a3737fa-5bf1-4d61-a683-7028764b81e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" Sep 18 16:58:17.228738 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:17.228699 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" Sep 18 16:58:17.543340 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:17.543311 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf"] Sep 18 16:58:17.546074 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:17.546043 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3737fa_5bf1_4d61_a683_7028764b81e3.slice/crio-a8580ccb1570d1fd62e3fa35365c9d27576c76a3674e047bfefd22f5fceee856 WatchSource:0}: Error finding container a8580ccb1570d1fd62e3fa35365c9d27576c76a3674e047bfefd22f5fceee856: Status 404 returned error can't find the container with id a8580ccb1570d1fd62e3fa35365c9d27576c76a3674e047bfefd22f5fceee856 Sep 18 16:58:18.459899 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:18.459856 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" event={"ID":"2a3737fa-5bf1-4d61-a683-7028764b81e3","Type":"ContainerStarted","Data":"a8580ccb1570d1fd62e3fa35365c9d27576c76a3674e047bfefd22f5fceee856"} Sep 18 16:58:21.474455 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:21.474394 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" event={"ID":"6dcbcdc3-43bf-4684-b600-7e967372b6ca","Type":"ContainerStarted","Data":"6fc170b4c5f5269e1d21b13154dc914940236ef7e286278f09eda8d3e02f2244"} Sep 18 16:58:21.474911 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:21.474598 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:21.476249 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:21.476226 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" event={"ID":"2a3737fa-5bf1-4d61-a683-7028764b81e3","Type":"ContainerStarted","Data":"05c968b7d1348de3d0da2b57ba58167122072705f7ab8f28fb2bc99012eb7677"} Sep 18 16:58:21.476357 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:21.476296 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" Sep 18 16:58:21.476357 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:21.476321 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" Sep 18 16:58:21.480501 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:21.480453 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" Sep 18 16:58:21.497351 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:21.497304 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffd896b95-vj7lp" podStartSLOduration=2.286709358 podStartE2EDuration="6.497290118s" totalImagesPullingTime="4.21058076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:21.495376418 +0000 UTC m=+274.829741125" watchObservedRunningTime="2026-09-18 16:58:21.497290118 +0000 UTC m=+274.831654825" Sep 18 16:58:21.513052 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:21.512997 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jckmf" podStartSLOduration=2.549272483 podStartE2EDuration="5.512980639s" totalImagesPullingTime="2.963708156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:21.512000212 +0000 UTC m=+274.846364922" watchObservedRunningTime="2026-09-18 16:58:21.512980639 +0000 UTC m=+274.847345345" Sep 18 16:58:26.454891 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:26.454857 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gklt9" Sep 18 16:58:30.801134 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.801095 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zqlqk"] Sep 18 16:58:30.804264 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.804241 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:30.806538 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.806516 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 18 16:58:30.806671 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.806653 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 18 16:58:30.806837 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.806819 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7xszz\"" Sep 18 16:58:30.806891 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.806879 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 18 16:58:30.807795 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.807777 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 18 16:58:30.941806 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.941773 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-textfile\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:30.941806 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.941818 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae82d90e-feef-4150-a881-418cc64351dd-sys\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:30.942070 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.941838 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-tls\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:30.942070 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.941891 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae82d90e-feef-4150-a881-418cc64351dd-metrics-client-ca\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:30.942070 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.941928 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae82d90e-feef-4150-a881-418cc64351dd-root\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:30.942070 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.941974 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:30.942070 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.942013 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2tvd\" (UniqueName: \"kubernetes.io/projected/ae82d90e-feef-4150-a881-418cc64351dd-kube-api-access-w2tvd\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:30.942070 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.942032 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:30.942070 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:30.942051 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-wtmp\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043069 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043022 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae82d90e-feef-4150-a881-418cc64351dd-sys\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043069 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043058 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-tls\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043069 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043078 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae82d90e-feef-4150-a881-418cc64351dd-metrics-client-ca\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043392 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043095 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae82d90e-feef-4150-a881-418cc64351dd-root\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043392 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043118 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043392 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043154 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2tvd\" (UniqueName: \"kubernetes.io/projected/ae82d90e-feef-4150-a881-418cc64351dd-kube-api-access-w2tvd\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043392 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043157 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae82d90e-feef-4150-a881-418cc64351dd-sys\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043392 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043181 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043392 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043190 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae82d90e-feef-4150-a881-418cc64351dd-root\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043392 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043224 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-wtmp\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043392 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043260 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-textfile\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043893 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043607 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-textfile\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043893 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043622 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-wtmp\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.043893 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.043781 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.044304 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.044080 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae82d90e-feef-4150-a881-418cc64351dd-metrics-client-ca\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.045682 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.045656 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-tls\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.045783 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.045684 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae82d90e-feef-4150-a881-418cc64351dd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.052895 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.052839 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2tvd\" (UniqueName: \"kubernetes.io/projected/ae82d90e-feef-4150-a881-418cc64351dd-kube-api-access-w2tvd\") pod \"node-exporter-zqlqk\" (UID: \"ae82d90e-feef-4150-a881-418cc64351dd\") " pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.113423 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.113375 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zqlqk" Sep 18 16:58:31.125672 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:58:31.125643 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae82d90e_feef_4150_a881_418cc64351dd.slice/crio-75b0bb43680f15abb8bd3c3d9db3745ecd2febd42bb48c06e4124c3e860e1da9 WatchSource:0}: Error finding container 75b0bb43680f15abb8bd3c3d9db3745ecd2febd42bb48c06e4124c3e860e1da9: Status 404 returned error can't find the container with id 75b0bb43680f15abb8bd3c3d9db3745ecd2febd42bb48c06e4124c3e860e1da9 Sep 18 16:58:31.443526 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.443493 2874 patch_prober.go:28] interesting pod/image-registry-56b5b5c5f4-tpsp6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 18 16:58:31.443713 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.443553 2874 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" podUID="5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 16:58:31.518022 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:31.517978 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zqlqk" event={"ID":"ae82d90e-feef-4150-a881-418cc64351dd","Type":"ContainerStarted","Data":"75b0bb43680f15abb8bd3c3d9db3745ecd2febd42bb48c06e4124c3e860e1da9"} Sep 18 16:58:32.523519 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:32.523386 2874 generic.go:356] "Generic (PLEG): container finished" podID="ae82d90e-feef-4150-a881-418cc64351dd" containerID="2c27f7173e0d41d8bf4c26b3802f200a5c8b4ebb4fa7c1c6b54af2faa634df7d" exitCode=0 Sep 18 16:58:32.523519 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:32.523478 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zqlqk" event={"ID":"ae82d90e-feef-4150-a881-418cc64351dd","Type":"ContainerDied","Data":"2c27f7173e0d41d8bf4c26b3802f200a5c8b4ebb4fa7c1c6b54af2faa634df7d"} Sep 18 16:58:33.529800 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:33.529763 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zqlqk" event={"ID":"ae82d90e-feef-4150-a881-418cc64351dd","Type":"ContainerStarted","Data":"61ae04a958b871ea157413e4150e9055e2ca2b1b299ba413fa71c8f6ee2c77a6"} Sep 18 16:58:33.530169 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:33.529823 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zqlqk" event={"ID":"ae82d90e-feef-4150-a881-418cc64351dd","Type":"ContainerStarted","Data":"b515cd0d9b8d7d11225efba87efb33140c120903fff7ccd091709a9302da200e"} Sep 18 16:58:33.552730 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:33.552673 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zqlqk" podStartSLOduration=2.437812977 podStartE2EDuration="3.552653523s" totalImagesPullingTime="1.042442297s" totalInitContainerRuntime="72.398249ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:33.551173637 +0000 UTC m=+286.885538346" watchObservedRunningTime="2026-09-18 16:58:33.552653523 +0000 UTC m=+286.887018231" Sep 18 16:58:33.853808 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:33.853724 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56b5b5c5f4-tpsp6"] Sep 18 16:58:33.857843 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:33.857822 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:58:38.392530 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:38.392496 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-npjd7" Sep 18 16:58:46.746383 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:46.746157 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 16:58:46.756757 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:46.746826 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 16:58:46.757284 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:46.757265 2874 kubelet.go:1648] "Image garbage collection succeeded" Sep 18 16:58:58.874096 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:58.874034 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" podUID="5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" containerName="registry" containerID="cri-o://e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be" gracePeriod=30 Sep 18 16:58:59.116231 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.116207 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:58:59.171612 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.171583 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-installation-pull-secrets\") pod \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " Sep 18 16:58:59.171807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.171622 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-certificates\") pod \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " Sep 18 16:58:59.171807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.171663 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-bound-sa-token\") pod \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " Sep 18 16:58:59.171807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.171694 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-trusted-ca\") pod \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " Sep 18 16:58:59.171807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.171714 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-ca-trust-extracted\") pod \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " Sep 18 16:58:59.171807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.171728 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") pod \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " Sep 18 16:58:59.171807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.171744 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4qsc\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-kube-api-access-h4qsc\") pod \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " Sep 18 16:58:59.171807 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.171778 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-image-registry-private-configuration\") pod \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\" (UID: \"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1\") " Sep 18 16:58:59.172258 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.172158 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:58:59.172258 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.172171 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:58:59.174347 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.174311 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:58:59.174532 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.174502 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-kube-api-access-h4qsc" (OuterVolumeSpecName: "kube-api-access-h4qsc") pod "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1"). InnerVolumeSpecName "kube-api-access-h4qsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:58:59.174532 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.174509 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:58:59.174648 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.174525 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:58:59.174708 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.174694 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:58:59.180133 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.180106 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" (UID: "5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:58:59.272461 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.272396 2874 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-bound-sa-token\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:59.272461 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.272456 2874 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-trusted-ca\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:59.272461 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.272464 2874 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-ca-trust-extracted\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:59.272461 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.272472 2874 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:59.272739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.272482 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4qsc\" (UniqueName: \"kubernetes.io/projected/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-kube-api-access-h4qsc\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:59.272739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.272492 2874 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-image-registry-private-configuration\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:59.272739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.272502 2874 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-installation-pull-secrets\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:59.272739 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.272510 2874 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1-registry-certificates\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 16:58:59.628332 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.628301 2874 generic.go:356] "Generic (PLEG): container finished" podID="5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" containerID="e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be" exitCode=0 Sep 18 16:58:59.628533 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.628347 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" Sep 18 16:58:59.628533 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.628392 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" event={"ID":"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1","Type":"ContainerDied","Data":"e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be"} Sep 18 16:58:59.628533 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.628488 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b5b5c5f4-tpsp6" event={"ID":"5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1","Type":"ContainerDied","Data":"6867561ada2cd1ad934824106fb4b8a15b4c1bb8f006111833e211d34b124c28"} Sep 18 16:58:59.628533 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.628509 2874 scope.go:117] "RemoveContainer" containerID="e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be" Sep 18 16:58:59.637527 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.637512 2874 scope.go:117] "RemoveContainer" containerID="e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be" Sep 18 16:58:59.637784 ip-10-0-132-99 kubenswrapper[2874]: E0918 16:58:59.637766 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be\": container with ID starting with e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be not found: ID does not exist" containerID="e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be" Sep 18 16:58:59.637822 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.637792 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be"} err="failed to get container status \"e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be\": rpc error: code = NotFound desc = could not find container \"e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be\": container with ID starting with e0e7a17ebfbdfa1933cf8746d10b78f4efdae6d38821b86b94b26eef30c063be not found: ID does not exist" Sep 18 16:58:59.654015 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.653990 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56b5b5c5f4-tpsp6"] Sep 18 16:58:59.658715 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:58:59.658693 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56b5b5c5f4-tpsp6"] Sep 18 16:59:00.747919 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:00.747880 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" path="/var/lib/kubelet/pods/5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1/volumes" Sep 18 16:59:13.683148 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:13.683114 2874 generic.go:356] "Generic (PLEG): container finished" podID="fea808a0-7a31-4e8c-a397-033dec7481ab" containerID="c7faa4f7adc711a32acb40b561ecfe05a66397d21a915060e9f2e85896a85142" exitCode=0 Sep 18 16:59:13.683594 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:13.683191 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" event={"ID":"fea808a0-7a31-4e8c-a397-033dec7481ab","Type":"ContainerDied","Data":"c7faa4f7adc711a32acb40b561ecfe05a66397d21a915060e9f2e85896a85142"} Sep 18 16:59:13.683594 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:13.683498 2874 scope.go:117] "RemoveContainer" containerID="c7faa4f7adc711a32acb40b561ecfe05a66397d21a915060e9f2e85896a85142" Sep 18 16:59:14.688571 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:14.688535 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kqbn6" event={"ID":"fea808a0-7a31-4e8c-a397-033dec7481ab","Type":"ContainerStarted","Data":"72efe1292d85089a193bd3934046dd8fd16ac694638416490bf1fef3835ce95e"} Sep 18 16:59:18.704818 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:18.704786 2874 generic.go:356] "Generic (PLEG): container finished" podID="59db72b3-cbc6-4634-bdcc-970021914be1" containerID="fd0eb86514e8d952072be8c24447304dcb60fc585e94c3c60d3afde8585dec43" exitCode=0 Sep 18 16:59:18.705228 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:18.704861 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" event={"ID":"59db72b3-cbc6-4634-bdcc-970021914be1","Type":"ContainerDied","Data":"fd0eb86514e8d952072be8c24447304dcb60fc585e94c3c60d3afde8585dec43"} Sep 18 16:59:18.705228 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:18.705213 2874 scope.go:117] "RemoveContainer" containerID="fd0eb86514e8d952072be8c24447304dcb60fc585e94c3c60d3afde8585dec43" Sep 18 16:59:18.706526 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:18.706507 2874 generic.go:356] "Generic (PLEG): container finished" podID="ea029ff9-8b9e-4e78-82dd-030bce98e106" containerID="6fd64f230fc6196b2529eb002774822cd54bbd5f5e9d316e271130883bde1ab2" exitCode=0 Sep 18 16:59:18.706595 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:18.706550 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" event={"ID":"ea029ff9-8b9e-4e78-82dd-030bce98e106","Type":"ContainerDied","Data":"6fd64f230fc6196b2529eb002774822cd54bbd5f5e9d316e271130883bde1ab2"} Sep 18 16:59:18.706808 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:18.706795 2874 scope.go:117] "RemoveContainer" containerID="6fd64f230fc6196b2529eb002774822cd54bbd5f5e9d316e271130883bde1ab2" Sep 18 16:59:19.712000 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:19.711964 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-4n4f7" event={"ID":"59db72b3-cbc6-4634-bdcc-970021914be1","Type":"ContainerStarted","Data":"b6c064e9ced2da602d2c32efc97f708b84999a010e69907bd2871c845596beb3"} Sep 18 16:59:19.713911 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:19.713884 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-phlgr" event={"ID":"ea029ff9-8b9e-4e78-82dd-030bce98e106","Type":"ContainerStarted","Data":"2138a28c4817623b11c8e80be833c6c6e776eadefa55f62979f5947608aec5b2"} Sep 18 16:59:55.195220 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.195184 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78667b4457-5bmgn"] Sep 18 16:59:55.195705 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.195478 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" containerName="registry" Sep 18 16:59:55.195705 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.195490 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" containerName="registry" Sep 18 16:59:55.195705 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.195552 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ebffbf1-54a9-4ec9-9e8e-130aeca4e7c1" containerName="registry" Sep 18 16:59:55.198547 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.198531 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.201760 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.201726 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 18 16:59:55.202241 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.202217 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 18 16:59:55.202427 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.202330 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hglbv\"" Sep 18 16:59:55.202427 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.202333 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 18 16:59:55.202427 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.202340 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 18 16:59:55.202684 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.202543 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 18 16:59:55.208359 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.208328 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 18 16:59:55.213827 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.213799 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78667b4457-5bmgn"] Sep 18 16:59:55.281840 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.281790 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-metrics-client-ca\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.281840 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.281838 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vtj\" (UniqueName: \"kubernetes.io/projected/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-kube-api-access-v6vtj\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.282055 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.281863 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-secret-telemeter-client\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.282055 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.281887 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.282055 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.281969 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-serving-certs-ca-bundle\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.282055 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.282020 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-telemeter-client-tls\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.282187 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.282068 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-federate-client-tls\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.282187 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.282089 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.383173 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.383132 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-telemeter-client-tls\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.383322 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.383179 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-federate-client-tls\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.383322 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.383207 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.383322 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.383243 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-metrics-client-ca\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.383532 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.383359 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vtj\" (UniqueName: \"kubernetes.io/projected/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-kube-api-access-v6vtj\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.383532 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.383438 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-secret-telemeter-client\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.383532 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.383467 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.383532 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.383522 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-serving-certs-ca-bundle\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.384254 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.384227 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-metrics-client-ca\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.384457 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.384426 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-serving-certs-ca-bundle\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.384580 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.384505 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.386058 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.386037 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-telemeter-client-tls\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.386165 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.386044 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.386165 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.386042 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-secret-telemeter-client\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.386289 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.386166 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-federate-client-tls\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.395192 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.395172 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vtj\" (UniqueName: \"kubernetes.io/projected/e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1-kube-api-access-v6vtj\") pod \"telemeter-client-78667b4457-5bmgn\" (UID: \"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1\") " pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.514004 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.513914 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" Sep 18 16:59:55.796043 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.796019 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78667b4457-5bmgn"] Sep 18 16:59:55.797775 ip-10-0-132-99 kubenswrapper[2874]: W0918 16:59:55.797753 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e2cbff_0ae0_4f2f_9b7a_8d9f6a07c2a1.slice/crio-ab4d074964c064b0215fb7a313a7fdf397ed6444bd5ac8f5921ffd79ac8224f0 WatchSource:0}: Error finding container ab4d074964c064b0215fb7a313a7fdf397ed6444bd5ac8f5921ffd79ac8224f0: Status 404 returned error can't find the container with id ab4d074964c064b0215fb7a313a7fdf397ed6444bd5ac8f5921ffd79ac8224f0 Sep 18 16:59:55.846566 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:55.846533 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" event={"ID":"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1","Type":"ContainerStarted","Data":"ab4d074964c064b0215fb7a313a7fdf397ed6444bd5ac8f5921ffd79ac8224f0"} Sep 18 16:59:57.856110 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:57.856075 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" event={"ID":"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1","Type":"ContainerStarted","Data":"ea54b37f46ab78fd317557dadd7736a2a6369ae343a54625591d8268f5e27245"} Sep 18 16:59:58.861477 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:58.861354 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" event={"ID":"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1","Type":"ContainerStarted","Data":"1acc501af089cec3f7e4c99b42b0bd206f53d19369ba2b77ac2138794757367d"} Sep 18 16:59:58.861477 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:58.861430 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" event={"ID":"e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1","Type":"ContainerStarted","Data":"5afb916a24f10b59bf02182081c92e09e44099308c9009206ac8692f3905bac5"} Sep 18 16:59:58.887851 ip-10-0-132-99 kubenswrapper[2874]: I0918 16:59:58.887809 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78667b4457-5bmgn" podStartSLOduration=1.287546943 podStartE2EDuration="3.887794807s" totalImagesPullingTime="2.600247864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:59:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:59:58.886606805 +0000 UTC m=+372.220971537" watchObservedRunningTime="2026-09-18 16:59:58.887794807 +0000 UTC m=+372.222159514" Sep 18 17:01:35.194793 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.194755 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg"] Sep 18 17:01:35.198079 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.198061 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.200590 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.200567 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 17:01:35.201905 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.201884 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 17:01:35.201905 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.201897 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hnskq\"" Sep 18 17:01:35.208215 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.208189 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg"] Sep 18 17:01:35.378719 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.378683 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.378895 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.378751 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.378895 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.378770 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdclx\" (UniqueName: \"kubernetes.io/projected/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-kube-api-access-gdclx\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.479890 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.479813 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.479890 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.479845 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdclx\" (UniqueName: \"kubernetes.io/projected/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-kube-api-access-gdclx\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.479890 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.479878 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.480232 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.480216 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.480272 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.480248 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.489356 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.489321 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdclx\" (UniqueName: \"kubernetes.io/projected/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-kube-api-access-gdclx\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.508271 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.508247 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:35.782958 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:35.782930 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg"] Sep 18 17:01:35.785664 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:01:35.785637 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acf9a24_d2d6_46d5_b359_e0c3f07532ff.slice/crio-1f5bad8b9b1c489c8689befa9fd346b4c6640961617395819a523247e9c70dca WatchSource:0}: Error finding container 1f5bad8b9b1c489c8689befa9fd346b4c6640961617395819a523247e9c70dca: Status 404 returned error can't find the container with id 1f5bad8b9b1c489c8689befa9fd346b4c6640961617395819a523247e9c70dca Sep 18 17:01:36.222207 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:36.222167 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" event={"ID":"3acf9a24-d2d6-46d5-b359-e0c3f07532ff","Type":"ContainerStarted","Data":"1f5bad8b9b1c489c8689befa9fd346b4c6640961617395819a523247e9c70dca"} Sep 18 17:01:42.250941 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:42.250909 2874 generic.go:356] "Generic (PLEG): container finished" podID="3acf9a24-d2d6-46d5-b359-e0c3f07532ff" containerID="b587e3567e26d60fdf12d16120a241f6319d53cf0ae2be19aca5a745aa10c12b" exitCode=0 Sep 18 17:01:42.251338 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:42.250973 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" event={"ID":"3acf9a24-d2d6-46d5-b359-e0c3f07532ff","Type":"ContainerDied","Data":"b587e3567e26d60fdf12d16120a241f6319d53cf0ae2be19aca5a745aa10c12b"} Sep 18 17:01:45.265348 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:45.265305 2874 generic.go:356] "Generic (PLEG): container finished" podID="3acf9a24-d2d6-46d5-b359-e0c3f07532ff" containerID="fb809134f64718be616eae458fb149207b858307c80cd30066e894cbebd6d655" exitCode=0 Sep 18 17:01:45.265770 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:45.265388 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" event={"ID":"3acf9a24-d2d6-46d5-b359-e0c3f07532ff","Type":"ContainerDied","Data":"fb809134f64718be616eae458fb149207b858307c80cd30066e894cbebd6d655"} Sep 18 17:01:52.298860 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:52.298825 2874 generic.go:356] "Generic (PLEG): container finished" podID="3acf9a24-d2d6-46d5-b359-e0c3f07532ff" containerID="6314633ca0727651bbee405cb372d351022ea36744443d2d4b3cb9ebf362aff5" exitCode=0 Sep 18 17:01:52.299268 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:52.298908 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" event={"ID":"3acf9a24-d2d6-46d5-b359-e0c3f07532ff","Type":"ContainerDied","Data":"6314633ca0727651bbee405cb372d351022ea36744443d2d4b3cb9ebf362aff5"} Sep 18 17:01:53.426654 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:53.426625 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:53.516289 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:53.516261 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-util\") pod \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " Sep 18 17:01:53.516480 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:53.516326 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdclx\" (UniqueName: \"kubernetes.io/projected/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-kube-api-access-gdclx\") pod \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " Sep 18 17:01:53.516480 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:53.516364 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-bundle\") pod \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\" (UID: \"3acf9a24-d2d6-46d5-b359-e0c3f07532ff\") " Sep 18 17:01:53.517129 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:53.517091 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-bundle" (OuterVolumeSpecName: "bundle") pod "3acf9a24-d2d6-46d5-b359-e0c3f07532ff" (UID: "3acf9a24-d2d6-46d5-b359-e0c3f07532ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:01:53.518666 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:53.518629 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-kube-api-access-gdclx" (OuterVolumeSpecName: "kube-api-access-gdclx") pod "3acf9a24-d2d6-46d5-b359-e0c3f07532ff" (UID: "3acf9a24-d2d6-46d5-b359-e0c3f07532ff"). InnerVolumeSpecName "kube-api-access-gdclx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:01:53.520340 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:53.520316 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-util" (OuterVolumeSpecName: "util") pod "3acf9a24-d2d6-46d5-b359-e0c3f07532ff" (UID: "3acf9a24-d2d6-46d5-b359-e0c3f07532ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:01:53.617203 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:53.617112 2874 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-util\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:01:53.617203 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:53.617154 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdclx\" (UniqueName: \"kubernetes.io/projected/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-kube-api-access-gdclx\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:01:53.617203 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:53.617166 2874 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3acf9a24-d2d6-46d5-b359-e0c3f07532ff-bundle\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:01:54.308711 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:54.308677 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" event={"ID":"3acf9a24-d2d6-46d5-b359-e0c3f07532ff","Type":"ContainerDied","Data":"1f5bad8b9b1c489c8689befa9fd346b4c6640961617395819a523247e9c70dca"} Sep 18 17:01:54.308878 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:54.308722 2874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f5bad8b9b1c489c8689befa9fd346b4c6640961617395819a523247e9c70dca" Sep 18 17:01:54.308878 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:54.308692 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4vzzg" Sep 18 17:01:58.377794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.377756 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh"] Sep 18 17:01:58.378214 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.378055 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acf9a24-d2d6-46d5-b359-e0c3f07532ff" containerName="pull" Sep 18 17:01:58.378214 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.378065 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acf9a24-d2d6-46d5-b359-e0c3f07532ff" containerName="pull" Sep 18 17:01:58.378214 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.378077 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acf9a24-d2d6-46d5-b359-e0c3f07532ff" containerName="extract" Sep 18 17:01:58.378214 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.378082 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acf9a24-d2d6-46d5-b359-e0c3f07532ff" containerName="extract" Sep 18 17:01:58.378214 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.378098 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acf9a24-d2d6-46d5-b359-e0c3f07532ff" containerName="util" Sep 18 17:01:58.378214 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.378104 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acf9a24-d2d6-46d5-b359-e0c3f07532ff" containerName="util" Sep 18 17:01:58.378214 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.378167 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="3acf9a24-d2d6-46d5-b359-e0c3f07532ff" containerName="extract" Sep 18 17:01:58.414703 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.414672 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh"] Sep 18 17:01:58.414848 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.414789 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" Sep 18 17:01:58.417779 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.417757 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 18 17:01:58.417779 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.417767 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 18 17:01:58.418011 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.417859 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-4h44b\"" Sep 18 17:01:58.418113 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.418098 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 18 17:01:58.453963 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.453936 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg2nl\" (UniqueName: \"kubernetes.io/projected/92ec18ff-0ef4-426e-8ffc-5a72787e1346-kube-api-access-dg2nl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4kksh\" (UID: \"92ec18ff-0ef4-426e-8ffc-5a72787e1346\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" Sep 18 17:01:58.454070 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.453985 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/92ec18ff-0ef4-426e-8ffc-5a72787e1346-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4kksh\" (UID: \"92ec18ff-0ef4-426e-8ffc-5a72787e1346\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" Sep 18 17:01:58.554856 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.554820 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/92ec18ff-0ef4-426e-8ffc-5a72787e1346-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4kksh\" (UID: \"92ec18ff-0ef4-426e-8ffc-5a72787e1346\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" Sep 18 17:01:58.555039 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.554897 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg2nl\" (UniqueName: \"kubernetes.io/projected/92ec18ff-0ef4-426e-8ffc-5a72787e1346-kube-api-access-dg2nl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4kksh\" (UID: \"92ec18ff-0ef4-426e-8ffc-5a72787e1346\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" Sep 18 17:01:58.557269 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.557246 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/92ec18ff-0ef4-426e-8ffc-5a72787e1346-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4kksh\" (UID: \"92ec18ff-0ef4-426e-8ffc-5a72787e1346\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" Sep 18 17:01:58.566256 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.566236 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg2nl\" (UniqueName: \"kubernetes.io/projected/92ec18ff-0ef4-426e-8ffc-5a72787e1346-kube-api-access-dg2nl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4kksh\" (UID: \"92ec18ff-0ef4-426e-8ffc-5a72787e1346\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" Sep 18 17:01:58.724560 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:58.724476 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" Sep 18 17:01:59.023266 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:59.023182 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh"] Sep 18 17:01:59.027839 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:01:59.027810 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ec18ff_0ef4_426e_8ffc_5a72787e1346.slice/crio-62898fd41899742425f5d9f526fea3a37cb3e7813bad76006a80774bdcfd2f38 WatchSource:0}: Error finding container 62898fd41899742425f5d9f526fea3a37cb3e7813bad76006a80774bdcfd2f38: Status 404 returned error can't find the container with id 62898fd41899742425f5d9f526fea3a37cb3e7813bad76006a80774bdcfd2f38 Sep 18 17:01:59.329826 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:01:59.329745 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" event={"ID":"92ec18ff-0ef4-426e-8ffc-5a72787e1346","Type":"ContainerStarted","Data":"62898fd41899742425f5d9f526fea3a37cb3e7813bad76006a80774bdcfd2f38"} Sep 18 17:02:03.349033 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.349004 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" event={"ID":"92ec18ff-0ef4-426e-8ffc-5a72787e1346","Type":"ContainerStarted","Data":"ec7d969e0cc901734e91393432e11862ad7fc37e9486cdc1da482e91ffd184f0"} Sep 18 17:02:03.349493 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.349095 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" Sep 18 17:02:03.378160 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.378117 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" podStartSLOduration=1.846979686 podStartE2EDuration="5.378102744s" totalImagesPullingTime="3.531123058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:02:03.376597724 +0000 UTC m=+496.710962430" watchObservedRunningTime="2026-09-18 17:02:03.378102744 +0000 UTC m=+496.712467451" Sep 18 17:02:03.511982 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.511947 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-rk775"] Sep 18 17:02:03.514185 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.514169 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:03.516723 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.516693 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Sep 18 17:02:03.516841 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.516810 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 18 17:02:03.516977 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.516961 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-9bd46\"" Sep 18 17:02:03.525866 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.525846 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-rk775"] Sep 18 17:02:03.595028 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.594991 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-certificates\") pod \"keda-operator-579c8958f7-rk775\" (UID: \"1bdec77f-6a6b-4b4b-87cd-3c0c81813412\") " pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:03.595198 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.595035 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qt8c\" (UniqueName: \"kubernetes.io/projected/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-kube-api-access-4qt8c\") pod \"keda-operator-579c8958f7-rk775\" (UID: \"1bdec77f-6a6b-4b4b-87cd-3c0c81813412\") " pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:03.595198 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.595100 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-cabundle0\") pod \"keda-operator-579c8958f7-rk775\" (UID: \"1bdec77f-6a6b-4b4b-87cd-3c0c81813412\") " pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:03.696467 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.696428 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-cabundle0\") pod \"keda-operator-579c8958f7-rk775\" (UID: \"1bdec77f-6a6b-4b4b-87cd-3c0c81813412\") " pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:03.696647 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.696514 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-certificates\") pod \"keda-operator-579c8958f7-rk775\" (UID: \"1bdec77f-6a6b-4b4b-87cd-3c0c81813412\") " pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:03.696647 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.696551 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qt8c\" (UniqueName: \"kubernetes.io/projected/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-kube-api-access-4qt8c\") pod \"keda-operator-579c8958f7-rk775\" (UID: \"1bdec77f-6a6b-4b4b-87cd-3c0c81813412\") " pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:03.696745 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:02:03.696672 2874 secret.go:281] references non-existent secret key: ca.crt Sep 18 17:02:03.696745 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:02:03.696694 2874 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Sep 18 17:02:03.696745 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:02:03.696703 2874 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-rk775: references non-existent secret key: ca.crt Sep 18 17:02:03.696852 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:02:03.696763 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-certificates podName:1bdec77f-6a6b-4b4b-87cd-3c0c81813412 nodeName:}" failed. No retries permitted until 2026-09-18 17:02:04.19674556 +0000 UTC m=+497.531110260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-certificates") pod "keda-operator-579c8958f7-rk775" (UID: "1bdec77f-6a6b-4b4b-87cd-3c0c81813412") : references non-existent secret key: ca.crt Sep 18 17:02:03.697137 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.697117 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-cabundle0\") pod \"keda-operator-579c8958f7-rk775\" (UID: \"1bdec77f-6a6b-4b4b-87cd-3c0c81813412\") " pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:03.715120 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.715087 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qt8c\" (UniqueName: \"kubernetes.io/projected/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-kube-api-access-4qt8c\") pod \"keda-operator-579c8958f7-rk775\" (UID: \"1bdec77f-6a6b-4b4b-87cd-3c0c81813412\") " pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:03.739977 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.739943 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk"] Sep 18 17:02:03.742845 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.742827 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:03.745376 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.745357 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Sep 18 17:02:03.756509 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.756488 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk"] Sep 18 17:02:03.796995 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.796975 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gmdhk\" (UID: \"fedc87fe-65f7-45f5-a518-cfe1e2b3675b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:03.797111 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.797044 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gmdhk\" (UID: \"fedc87fe-65f7-45f5-a518-cfe1e2b3675b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:03.797111 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.797066 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968jp\" (UniqueName: \"kubernetes.io/projected/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-kube-api-access-968jp\") pod \"keda-metrics-apiserver-77857b7b98-gmdhk\" (UID: \"fedc87fe-65f7-45f5-a518-cfe1e2b3675b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:03.898125 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.898092 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gmdhk\" (UID: \"fedc87fe-65f7-45f5-a518-cfe1e2b3675b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:03.898125 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.898130 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968jp\" (UniqueName: \"kubernetes.io/projected/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-kube-api-access-968jp\") pod \"keda-metrics-apiserver-77857b7b98-gmdhk\" (UID: \"fedc87fe-65f7-45f5-a518-cfe1e2b3675b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:03.898377 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.898154 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gmdhk\" (UID: \"fedc87fe-65f7-45f5-a518-cfe1e2b3675b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:03.898377 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:02:03.898266 2874 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found Sep 18 17:02:03.898377 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:02:03.898297 2874 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk: secret "keda-metrics-apiserver-certs" not found Sep 18 17:02:03.898377 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:02:03.898376 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-certificates podName:fedc87fe-65f7-45f5-a518-cfe1e2b3675b nodeName:}" failed. No retries permitted until 2026-09-18 17:02:04.398355299 +0000 UTC m=+497.732720003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-certificates") pod "keda-metrics-apiserver-77857b7b98-gmdhk" (UID: "fedc87fe-65f7-45f5-a518-cfe1e2b3675b") : secret "keda-metrics-apiserver-certs" not found Sep 18 17:02:03.898650 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.898511 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gmdhk\" (UID: \"fedc87fe-65f7-45f5-a518-cfe1e2b3675b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:03.908579 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:03.908559 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-968jp\" (UniqueName: \"kubernetes.io/projected/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-kube-api-access-968jp\") pod \"keda-metrics-apiserver-77857b7b98-gmdhk\" (UID: \"fedc87fe-65f7-45f5-a518-cfe1e2b3675b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:04.201311 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:04.201276 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-certificates\") pod \"keda-operator-579c8958f7-rk775\" (UID: \"1bdec77f-6a6b-4b4b-87cd-3c0c81813412\") " pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:04.203835 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:04.203810 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1bdec77f-6a6b-4b4b-87cd-3c0c81813412-certificates\") pod \"keda-operator-579c8958f7-rk775\" (UID: \"1bdec77f-6a6b-4b4b-87cd-3c0c81813412\") " pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:04.403931 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:04.403890 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gmdhk\" (UID: \"fedc87fe-65f7-45f5-a518-cfe1e2b3675b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:04.406562 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:04.406528 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fedc87fe-65f7-45f5-a518-cfe1e2b3675b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gmdhk\" (UID: \"fedc87fe-65f7-45f5-a518-cfe1e2b3675b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:04.424332 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:04.424301 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:04.653595 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:04.653560 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:04.720395 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:04.720361 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-rk775"] Sep 18 17:02:04.723184 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:02:04.723157 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdec77f_6a6b_4b4b_87cd_3c0c81813412.slice/crio-c59cb952f13b7b3b691d78d10a432f09e5f157de636c7101edd99b08a1bd9c54 WatchSource:0}: Error finding container c59cb952f13b7b3b691d78d10a432f09e5f157de636c7101edd99b08a1bd9c54: Status 404 returned error can't find the container with id c59cb952f13b7b3b691d78d10a432f09e5f157de636c7101edd99b08a1bd9c54 Sep 18 17:02:04.725056 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:04.725035 2874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:02:04.925601 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:04.925577 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk"] Sep 18 17:02:04.928212 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:02:04.928182 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfedc87fe_65f7_45f5_a518_cfe1e2b3675b.slice/crio-04d88c3e17a5081a98efa6d1906a22ce3ae50aa2117bb5f8d90639c40050f3c3 WatchSource:0}: Error finding container 04d88c3e17a5081a98efa6d1906a22ce3ae50aa2117bb5f8d90639c40050f3c3: Status 404 returned error can't find the container with id 04d88c3e17a5081a98efa6d1906a22ce3ae50aa2117bb5f8d90639c40050f3c3 Sep 18 17:02:05.358823 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:05.358789 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-rk775" event={"ID":"1bdec77f-6a6b-4b4b-87cd-3c0c81813412","Type":"ContainerStarted","Data":"c59cb952f13b7b3b691d78d10a432f09e5f157de636c7101edd99b08a1bd9c54"} Sep 18 17:02:05.359950 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:05.359926 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" event={"ID":"fedc87fe-65f7-45f5-a518-cfe1e2b3675b","Type":"ContainerStarted","Data":"04d88c3e17a5081a98efa6d1906a22ce3ae50aa2117bb5f8d90639c40050f3c3"} Sep 18 17:02:09.381686 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:09.381640 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" event={"ID":"fedc87fe-65f7-45f5-a518-cfe1e2b3675b","Type":"ContainerStarted","Data":"25f57508d6d41803f7d8480d2db353040c9fb543dad9ea816d92dc472d0648ff"} Sep 18 17:02:09.382158 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:09.381721 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:09.383441 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:09.383389 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-rk775" event={"ID":"1bdec77f-6a6b-4b4b-87cd-3c0c81813412","Type":"ContainerStarted","Data":"a37aed4fbfd0946d8d45b80dded95a2d157c1a94fac3f2f3b281cc6b59efc68d"} Sep 18 17:02:09.383547 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:09.383474 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:02:09.404845 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:09.404803 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" podStartSLOduration=2.6002207090000002 podStartE2EDuration="6.404790237s" totalImagesPullingTime="3.804569528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:02:09.403180977 +0000 UTC m=+502.737545725" watchObservedRunningTime="2026-09-18 17:02:09.404790237 +0000 UTC m=+502.739154944" Sep 18 17:02:09.423652 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:09.423602 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-rk775" podStartSLOduration=2.409269611 podStartE2EDuration="6.423588908s" totalImagesPullingTime="4.014319297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:02:09.422368186 +0000 UTC m=+502.756732905" watchObservedRunningTime="2026-09-18 17:02:09.423588908 +0000 UTC m=+502.757953612" Sep 18 17:02:20.391662 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:20.391631 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gmdhk" Sep 18 17:02:24.356225 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:24.356193 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4kksh" Sep 18 17:02:30.389470 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:02:30.389440 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-rk775" Sep 18 17:03:25.090663 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.090583 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c47bdc49c-shwkm"] Sep 18 17:03:25.093370 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.093350 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:03:25.096427 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.096386 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 18 17:03:25.096547 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.096435 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 18 17:03:25.096547 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.096449 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Sep 18 17:03:25.096547 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.096460 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-l86mv\"" Sep 18 17:03:25.106716 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.106691 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c47bdc49c-shwkm"] Sep 18 17:03:25.136150 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.136123 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-pz86p"] Sep 18 17:03:25.138648 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.138616 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-pz86p" Sep 18 17:03:25.141335 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.141314 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 18 17:03:25.141455 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.141425 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8nb7\"" Sep 18 17:03:25.152323 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.152300 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-pz86p"] Sep 18 17:03:25.168349 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.168320 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pgn\" (UniqueName: \"kubernetes.io/projected/e500bc99-4e31-44a0-a0f1-4f4962638c18-kube-api-access-n7pgn\") pod \"kserve-controller-manager-5c47bdc49c-shwkm\" (UID: \"e500bc99-4e31-44a0-a0f1-4f4962638c18\") " pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:03:25.168499 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.168371 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e500bc99-4e31-44a0-a0f1-4f4962638c18-cert\") pod \"kserve-controller-manager-5c47bdc49c-shwkm\" (UID: \"e500bc99-4e31-44a0-a0f1-4f4962638c18\") " pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:03:25.269389 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.269357 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e500bc99-4e31-44a0-a0f1-4f4962638c18-cert\") pod \"kserve-controller-manager-5c47bdc49c-shwkm\" (UID: \"e500bc99-4e31-44a0-a0f1-4f4962638c18\") " pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:03:25.269554 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.269482 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mwtv\" (UniqueName: \"kubernetes.io/projected/75500dae-e5dd-48cb-915b-c796da2401ba-kube-api-access-5mwtv\") pod \"seaweedfs-64568bcd49-pz86p\" (UID: \"75500dae-e5dd-48cb-915b-c796da2401ba\") " pod="kserve/seaweedfs-64568bcd49-pz86p" Sep 18 17:03:25.269554 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:03:25.269505 2874 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 18 17:03:25.269554 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.269522 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75500dae-e5dd-48cb-915b-c796da2401ba-data\") pod \"seaweedfs-64568bcd49-pz86p\" (UID: \"75500dae-e5dd-48cb-915b-c796da2401ba\") " pod="kserve/seaweedfs-64568bcd49-pz86p" Sep 18 17:03:25.269713 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.269553 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pgn\" (UniqueName: \"kubernetes.io/projected/e500bc99-4e31-44a0-a0f1-4f4962638c18-kube-api-access-n7pgn\") pod \"kserve-controller-manager-5c47bdc49c-shwkm\" (UID: \"e500bc99-4e31-44a0-a0f1-4f4962638c18\") " pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:03:25.269713 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:03:25.269568 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e500bc99-4e31-44a0-a0f1-4f4962638c18-cert podName:e500bc99-4e31-44a0-a0f1-4f4962638c18 nodeName:}" failed. No retries permitted until 2026-09-18 17:03:25.769554304 +0000 UTC m=+579.103918989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e500bc99-4e31-44a0-a0f1-4f4962638c18-cert") pod "kserve-controller-manager-5c47bdc49c-shwkm" (UID: "e500bc99-4e31-44a0-a0f1-4f4962638c18") : secret "kserve-webhook-server-cert" not found Sep 18 17:03:25.280896 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.280871 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pgn\" (UniqueName: \"kubernetes.io/projected/e500bc99-4e31-44a0-a0f1-4f4962638c18-kube-api-access-n7pgn\") pod \"kserve-controller-manager-5c47bdc49c-shwkm\" (UID: \"e500bc99-4e31-44a0-a0f1-4f4962638c18\") " pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:03:25.370654 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.370572 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mwtv\" (UniqueName: \"kubernetes.io/projected/75500dae-e5dd-48cb-915b-c796da2401ba-kube-api-access-5mwtv\") pod \"seaweedfs-64568bcd49-pz86p\" (UID: \"75500dae-e5dd-48cb-915b-c796da2401ba\") " pod="kserve/seaweedfs-64568bcd49-pz86p" Sep 18 17:03:25.370654 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.370608 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75500dae-e5dd-48cb-915b-c796da2401ba-data\") pod \"seaweedfs-64568bcd49-pz86p\" (UID: \"75500dae-e5dd-48cb-915b-c796da2401ba\") " pod="kserve/seaweedfs-64568bcd49-pz86p" Sep 18 17:03:25.370930 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.370910 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75500dae-e5dd-48cb-915b-c796da2401ba-data\") pod \"seaweedfs-64568bcd49-pz86p\" (UID: \"75500dae-e5dd-48cb-915b-c796da2401ba\") " pod="kserve/seaweedfs-64568bcd49-pz86p" Sep 18 17:03:25.384724 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.384707 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mwtv\" (UniqueName: \"kubernetes.io/projected/75500dae-e5dd-48cb-915b-c796da2401ba-kube-api-access-5mwtv\") pod \"seaweedfs-64568bcd49-pz86p\" (UID: \"75500dae-e5dd-48cb-915b-c796da2401ba\") " pod="kserve/seaweedfs-64568bcd49-pz86p" Sep 18 17:03:25.446937 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.446899 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-pz86p" Sep 18 17:03:25.718792 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.718759 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-pz86p"] Sep 18 17:03:25.721475 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:03:25.721445 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75500dae_e5dd_48cb_915b_c796da2401ba.slice/crio-674921b0c0c7dff2a35b71063a1002014661294ad0d91178fd36e3cd9f024380 WatchSource:0}: Error finding container 674921b0c0c7dff2a35b71063a1002014661294ad0d91178fd36e3cd9f024380: Status 404 returned error can't find the container with id 674921b0c0c7dff2a35b71063a1002014661294ad0d91178fd36e3cd9f024380 Sep 18 17:03:25.773760 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.773720 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e500bc99-4e31-44a0-a0f1-4f4962638c18-cert\") pod \"kserve-controller-manager-5c47bdc49c-shwkm\" (UID: \"e500bc99-4e31-44a0-a0f1-4f4962638c18\") " pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:03:25.776341 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:25.776312 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e500bc99-4e31-44a0-a0f1-4f4962638c18-cert\") pod \"kserve-controller-manager-5c47bdc49c-shwkm\" (UID: \"e500bc99-4e31-44a0-a0f1-4f4962638c18\") " pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:03:26.025192 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:26.025113 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:03:26.297478 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:26.297450 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c47bdc49c-shwkm"] Sep 18 17:03:26.340998 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:03:26.340966 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode500bc99_4e31_44a0_a0f1_4f4962638c18.slice/crio-88408d75932d9a5109baf7b6e087fa9512573e72d01854806c434b4a14213128 WatchSource:0}: Error finding container 88408d75932d9a5109baf7b6e087fa9512573e72d01854806c434b4a14213128: Status 404 returned error can't find the container with id 88408d75932d9a5109baf7b6e087fa9512573e72d01854806c434b4a14213128 Sep 18 17:03:26.730806 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:26.730769 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-pz86p" event={"ID":"75500dae-e5dd-48cb-915b-c796da2401ba","Type":"ContainerStarted","Data":"674921b0c0c7dff2a35b71063a1002014661294ad0d91178fd36e3cd9f024380"} Sep 18 17:03:26.732206 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:26.732166 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" event={"ID":"e500bc99-4e31-44a0-a0f1-4f4962638c18","Type":"ContainerStarted","Data":"88408d75932d9a5109baf7b6e087fa9512573e72d01854806c434b4a14213128"} Sep 18 17:03:30.753849 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:30.753807 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-pz86p" event={"ID":"75500dae-e5dd-48cb-915b-c796da2401ba","Type":"ContainerStarted","Data":"29ce52a0e078ad1e58537b37aa67bb8aea98a8021cb37a46362cd29e56c31fbb"} Sep 18 17:03:30.754367 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:30.753871 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-pz86p" Sep 18 17:03:30.755346 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:30.755323 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" event={"ID":"e500bc99-4e31-44a0-a0f1-4f4962638c18","Type":"ContainerStarted","Data":"bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a"} Sep 18 17:03:30.755483 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:30.755438 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:03:30.772262 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:30.772209 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-pz86p" podStartSLOduration=1.579940273 podStartE2EDuration="5.772191819s" totalImagesPullingTime="4.192251546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:03:30.772026695 +0000 UTC m=+584.106391401" watchObservedRunningTime="2026-09-18 17:03:30.772191819 +0000 UTC m=+584.106556523" Sep 18 17:03:30.790050 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:30.790004 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" podStartSLOduration=2.319867956 podStartE2EDuration="5.789989871s" totalImagesPullingTime="3.470121915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:03:30.789253215 +0000 UTC m=+584.123617922" watchObservedRunningTime="2026-09-18 17:03:30.789989871 +0000 UTC m=+584.124354774" Sep 18 17:03:36.762678 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:36.762647 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-pz86p" Sep 18 17:03:46.777751 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:46.777647 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:03:46.780788 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:03:46.778622 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:04:01.765313 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:01.765280 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:04:09.097874 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.097841 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c47bdc49c-shwkm"] Sep 18 17:04:09.098296 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.098038 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" podUID="e500bc99-4e31-44a0-a0f1-4f4962638c18" containerName="manager" containerID="cri-o://bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a" gracePeriod=10 Sep 18 17:04:09.121226 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.121201 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c47bdc49c-nrnp6"] Sep 18 17:04:09.123580 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.123565 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" Sep 18 17:04:09.134702 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.134679 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c47bdc49c-nrnp6"] Sep 18 17:04:09.238034 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.237998 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c27145e9-a898-4af7-96de-7d3da1bc20c7-cert\") pod \"kserve-controller-manager-5c47bdc49c-nrnp6\" (UID: \"c27145e9-a898-4af7-96de-7d3da1bc20c7\") " pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" Sep 18 17:04:09.238173 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.238131 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kb4\" (UniqueName: \"kubernetes.io/projected/c27145e9-a898-4af7-96de-7d3da1bc20c7-kube-api-access-x4kb4\") pod \"kserve-controller-manager-5c47bdc49c-nrnp6\" (UID: \"c27145e9-a898-4af7-96de-7d3da1bc20c7\") " pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" Sep 18 17:04:09.339230 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.339204 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:04:09.339340 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.339207 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c27145e9-a898-4af7-96de-7d3da1bc20c7-cert\") pod \"kserve-controller-manager-5c47bdc49c-nrnp6\" (UID: \"c27145e9-a898-4af7-96de-7d3da1bc20c7\") " pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" Sep 18 17:04:09.339393 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.339349 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kb4\" (UniqueName: \"kubernetes.io/projected/c27145e9-a898-4af7-96de-7d3da1bc20c7-kube-api-access-x4kb4\") pod \"kserve-controller-manager-5c47bdc49c-nrnp6\" (UID: \"c27145e9-a898-4af7-96de-7d3da1bc20c7\") " pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" Sep 18 17:04:09.341566 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.341547 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c27145e9-a898-4af7-96de-7d3da1bc20c7-cert\") pod \"kserve-controller-manager-5c47bdc49c-nrnp6\" (UID: \"c27145e9-a898-4af7-96de-7d3da1bc20c7\") " pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" Sep 18 17:04:09.348525 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.348474 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kb4\" (UniqueName: \"kubernetes.io/projected/c27145e9-a898-4af7-96de-7d3da1bc20c7-kube-api-access-x4kb4\") pod \"kserve-controller-manager-5c47bdc49c-nrnp6\" (UID: \"c27145e9-a898-4af7-96de-7d3da1bc20c7\") " pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" Sep 18 17:04:09.440477 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.440438 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e500bc99-4e31-44a0-a0f1-4f4962638c18-cert\") pod \"e500bc99-4e31-44a0-a0f1-4f4962638c18\" (UID: \"e500bc99-4e31-44a0-a0f1-4f4962638c18\") " Sep 18 17:04:09.440477 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.440497 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7pgn\" (UniqueName: \"kubernetes.io/projected/e500bc99-4e31-44a0-a0f1-4f4962638c18-kube-api-access-n7pgn\") pod \"e500bc99-4e31-44a0-a0f1-4f4962638c18\" (UID: \"e500bc99-4e31-44a0-a0f1-4f4962638c18\") " Sep 18 17:04:09.442708 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.442678 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e500bc99-4e31-44a0-a0f1-4f4962638c18-kube-api-access-n7pgn" (OuterVolumeSpecName: "kube-api-access-n7pgn") pod "e500bc99-4e31-44a0-a0f1-4f4962638c18" (UID: "e500bc99-4e31-44a0-a0f1-4f4962638c18"). InnerVolumeSpecName "kube-api-access-n7pgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:04:09.442813 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.442703 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e500bc99-4e31-44a0-a0f1-4f4962638c18-cert" (OuterVolumeSpecName: "cert") pod "e500bc99-4e31-44a0-a0f1-4f4962638c18" (UID: "e500bc99-4e31-44a0-a0f1-4f4962638c18"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:04:09.482883 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.482859 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" Sep 18 17:04:09.541637 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.541607 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7pgn\" (UniqueName: \"kubernetes.io/projected/e500bc99-4e31-44a0-a0f1-4f4962638c18-kube-api-access-n7pgn\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:04:09.541765 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.541642 2874 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e500bc99-4e31-44a0-a0f1-4f4962638c18-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:04:09.756061 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.756036 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c47bdc49c-nrnp6"] Sep 18 17:04:09.758257 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:04:09.758232 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27145e9_a898_4af7_96de_7d3da1bc20c7.slice/crio-a1bbb201f68f611bfddc4b2f9fa761926a782440ac5fcfa61727acd90bd2579b WatchSource:0}: Error finding container a1bbb201f68f611bfddc4b2f9fa761926a782440ac5fcfa61727acd90bd2579b: Status 404 returned error can't find the container with id a1bbb201f68f611bfddc4b2f9fa761926a782440ac5fcfa61727acd90bd2579b Sep 18 17:04:09.923836 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.923799 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" event={"ID":"c27145e9-a898-4af7-96de-7d3da1bc20c7","Type":"ContainerStarted","Data":"a1bbb201f68f611bfddc4b2f9fa761926a782440ac5fcfa61727acd90bd2579b"} Sep 18 17:04:09.925168 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.925141 2874 generic.go:356] "Generic (PLEG): container finished" podID="e500bc99-4e31-44a0-a0f1-4f4962638c18" containerID="bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a" exitCode=0 Sep 18 17:04:09.925293 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.925185 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" event={"ID":"e500bc99-4e31-44a0-a0f1-4f4962638c18","Type":"ContainerDied","Data":"bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a"} Sep 18 17:04:09.925293 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.925218 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" event={"ID":"e500bc99-4e31-44a0-a0f1-4f4962638c18","Type":"ContainerDied","Data":"88408d75932d9a5109baf7b6e087fa9512573e72d01854806c434b4a14213128"} Sep 18 17:04:09.925293 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.925187 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c47bdc49c-shwkm" Sep 18 17:04:09.925293 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.925267 2874 scope.go:117] "RemoveContainer" containerID="bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a" Sep 18 17:04:09.934794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.934774 2874 scope.go:117] "RemoveContainer" containerID="bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a" Sep 18 17:04:09.935083 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:04:09.935061 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a\": container with ID starting with bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a not found: ID does not exist" containerID="bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a" Sep 18 17:04:09.935152 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.935097 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a"} err="failed to get container status \"bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a\": rpc error: code = NotFound desc = could not find container \"bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a\": container with ID starting with bb80167bae604772ac57fa7f01dbc4c6ccdeeecdef61fca3157b4f350400e16a not found: ID does not exist" Sep 18 17:04:09.949233 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.949213 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c47bdc49c-shwkm"] Sep 18 17:04:09.953821 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:09.953797 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5c47bdc49c-shwkm"] Sep 18 17:04:10.749192 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:10.749160 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e500bc99-4e31-44a0-a0f1-4f4962638c18" path="/var/lib/kubelet/pods/e500bc99-4e31-44a0-a0f1-4f4962638c18/volumes" Sep 18 17:04:10.932295 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:10.932259 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" event={"ID":"c27145e9-a898-4af7-96de-7d3da1bc20c7","Type":"ContainerStarted","Data":"c1cf72e54b11ce09020ebd873bf8840c1e2d28efa75a5a3224e4dd933802e5ea"} Sep 18 17:04:10.932460 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:10.932350 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" Sep 18 17:04:10.951332 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:10.951292 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" podStartSLOduration=1.632147725 podStartE2EDuration="1.951278246s" totalImagesPullingTime="319.130521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:04:10.949704425 +0000 UTC m=+624.284069146" watchObservedRunningTime="2026-09-18 17:04:10.951278246 +0000 UTC m=+624.285643011" Sep 18 17:04:41.942889 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:41.942848 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c47bdc49c-nrnp6" Sep 18 17:04:43.409972 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.409936 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-v8nnn"] Sep 18 17:04:43.410388 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.410253 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e500bc99-4e31-44a0-a0f1-4f4962638c18" containerName="manager" Sep 18 17:04:43.410388 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.410290 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="e500bc99-4e31-44a0-a0f1-4f4962638c18" containerName="manager" Sep 18 17:04:43.410388 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.410366 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="e500bc99-4e31-44a0-a0f1-4f4962638c18" containerName="manager" Sep 18 17:04:43.413021 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.413003 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:04:43.414035 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.414005 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-gpwpm"] Sep 18 17:04:43.417001 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.416985 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:43.421234 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.421219 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Sep 18 17:04:43.421313 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.421297 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-lkv5q\"" Sep 18 17:04:43.422524 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.422508 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Sep 18 17:04:43.422601 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.422524 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-6tprj\"" Sep 18 17:04:43.422653 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.422605 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\"" Sep 18 17:04:43.441206 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.441166 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-v8nnn"] Sep 18 17:04:43.443891 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.443872 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-gpwpm"] Sep 18 17:04:43.523990 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.523958 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88eaa60c-05b5-4f53-8212-2e86a4b4e897-metrics-certs\") pod \"odh-model-controller-7b966d7984-gpwpm\" (UID: \"88eaa60c-05b5-4f53-8212-2e86a4b4e897\") " pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:43.523990 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.523990 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b057e4-8030-4311-a307-fb2f16b3d35a-tls-certs\") pod \"model-serving-api-7b954ffc95-v8nnn\" (UID: \"10b057e4-8030-4311-a307-fb2f16b3d35a\") " pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:04:43.524186 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.524013 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psfpr\" (UniqueName: \"kubernetes.io/projected/10b057e4-8030-4311-a307-fb2f16b3d35a-kube-api-access-psfpr\") pod \"model-serving-api-7b954ffc95-v8nnn\" (UID: \"10b057e4-8030-4311-a307-fb2f16b3d35a\") " pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:04:43.524186 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.524094 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflwt\" (UniqueName: \"kubernetes.io/projected/88eaa60c-05b5-4f53-8212-2e86a4b4e897-kube-api-access-rflwt\") pod \"odh-model-controller-7b966d7984-gpwpm\" (UID: \"88eaa60c-05b5-4f53-8212-2e86a4b4e897\") " pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:43.524186 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.524139 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88eaa60c-05b5-4f53-8212-2e86a4b4e897-cert\") pod \"odh-model-controller-7b966d7984-gpwpm\" (UID: \"88eaa60c-05b5-4f53-8212-2e86a4b4e897\") " pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:43.625342 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.625296 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88eaa60c-05b5-4f53-8212-2e86a4b4e897-metrics-certs\") pod \"odh-model-controller-7b966d7984-gpwpm\" (UID: \"88eaa60c-05b5-4f53-8212-2e86a4b4e897\") " pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:43.625342 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.625337 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b057e4-8030-4311-a307-fb2f16b3d35a-tls-certs\") pod \"model-serving-api-7b954ffc95-v8nnn\" (UID: \"10b057e4-8030-4311-a307-fb2f16b3d35a\") " pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:04:43.625635 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.625362 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psfpr\" (UniqueName: \"kubernetes.io/projected/10b057e4-8030-4311-a307-fb2f16b3d35a-kube-api-access-psfpr\") pod \"model-serving-api-7b954ffc95-v8nnn\" (UID: \"10b057e4-8030-4311-a307-fb2f16b3d35a\") " pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:04:43.625635 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.625432 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflwt\" (UniqueName: \"kubernetes.io/projected/88eaa60c-05b5-4f53-8212-2e86a4b4e897-kube-api-access-rflwt\") pod \"odh-model-controller-7b966d7984-gpwpm\" (UID: \"88eaa60c-05b5-4f53-8212-2e86a4b4e897\") " pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:43.625635 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.625475 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88eaa60c-05b5-4f53-8212-2e86a4b4e897-cert\") pod \"odh-model-controller-7b966d7984-gpwpm\" (UID: \"88eaa60c-05b5-4f53-8212-2e86a4b4e897\") " pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:43.625635 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:04:43.625504 2874 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 18 17:04:43.625635 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:04:43.625573 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b057e4-8030-4311-a307-fb2f16b3d35a-tls-certs podName:10b057e4-8030-4311-a307-fb2f16b3d35a nodeName:}" failed. No retries permitted until 2026-09-18 17:04:44.125552987 +0000 UTC m=+657.459917687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/10b057e4-8030-4311-a307-fb2f16b3d35a-tls-certs") pod "model-serving-api-7b954ffc95-v8nnn" (UID: "10b057e4-8030-4311-a307-fb2f16b3d35a") : secret "model-serving-api-tls" not found Sep 18 17:04:43.625635 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:04:43.625586 2874 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 18 17:04:43.625635 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:04:43.625630 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88eaa60c-05b5-4f53-8212-2e86a4b4e897-cert podName:88eaa60c-05b5-4f53-8212-2e86a4b4e897 nodeName:}" failed. No retries permitted until 2026-09-18 17:04:44.125619012 +0000 UTC m=+657.459983697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/88eaa60c-05b5-4f53-8212-2e86a4b4e897-cert") pod "odh-model-controller-7b966d7984-gpwpm" (UID: "88eaa60c-05b5-4f53-8212-2e86a4b4e897") : secret "odh-model-controller-webhook-cert" not found Sep 18 17:04:43.627972 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.627944 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88eaa60c-05b5-4f53-8212-2e86a4b4e897-metrics-certs\") pod \"odh-model-controller-7b966d7984-gpwpm\" (UID: \"88eaa60c-05b5-4f53-8212-2e86a4b4e897\") " pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:43.656705 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.656677 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflwt\" (UniqueName: \"kubernetes.io/projected/88eaa60c-05b5-4f53-8212-2e86a4b4e897-kube-api-access-rflwt\") pod \"odh-model-controller-7b966d7984-gpwpm\" (UID: \"88eaa60c-05b5-4f53-8212-2e86a4b4e897\") " pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:43.658145 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:43.658121 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psfpr\" (UniqueName: \"kubernetes.io/projected/10b057e4-8030-4311-a307-fb2f16b3d35a-kube-api-access-psfpr\") pod \"model-serving-api-7b954ffc95-v8nnn\" (UID: \"10b057e4-8030-4311-a307-fb2f16b3d35a\") " pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:04:44.130703 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:44.130666 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88eaa60c-05b5-4f53-8212-2e86a4b4e897-cert\") pod \"odh-model-controller-7b966d7984-gpwpm\" (UID: \"88eaa60c-05b5-4f53-8212-2e86a4b4e897\") " pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:44.130933 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:44.130773 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b057e4-8030-4311-a307-fb2f16b3d35a-tls-certs\") pod \"model-serving-api-7b954ffc95-v8nnn\" (UID: \"10b057e4-8030-4311-a307-fb2f16b3d35a\") " pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:04:44.133474 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:44.133447 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b057e4-8030-4311-a307-fb2f16b3d35a-tls-certs\") pod \"model-serving-api-7b954ffc95-v8nnn\" (UID: \"10b057e4-8030-4311-a307-fb2f16b3d35a\") " pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:04:44.133608 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:44.133592 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88eaa60c-05b5-4f53-8212-2e86a4b4e897-cert\") pod \"odh-model-controller-7b966d7984-gpwpm\" (UID: \"88eaa60c-05b5-4f53-8212-2e86a4b4e897\") " pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:44.324778 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:44.324724 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:04:44.331770 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:44.331749 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:44.608721 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:44.608698 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-v8nnn"] Sep 18 17:04:44.610824 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:04:44.610801 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b057e4_8030_4311_a307_fb2f16b3d35a.slice/crio-b7aa1b80896528f11b92230afeadb30e900c98c33a6aef876f67f829546a3880 WatchSource:0}: Error finding container b7aa1b80896528f11b92230afeadb30e900c98c33a6aef876f67f829546a3880: Status 404 returned error can't find the container with id b7aa1b80896528f11b92230afeadb30e900c98c33a6aef876f67f829546a3880 Sep 18 17:04:44.626613 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:44.626588 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-gpwpm"] Sep 18 17:04:44.627646 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:04:44.627622 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eaa60c_05b5_4f53_8212_2e86a4b4e897.slice/crio-07d88f4ffd8f74d66746d24e3790f5a1f024ee766c62392240b69a9e1a11ab9e WatchSource:0}: Error finding container 07d88f4ffd8f74d66746d24e3790f5a1f024ee766c62392240b69a9e1a11ab9e: Status 404 returned error can't find the container with id 07d88f4ffd8f74d66746d24e3790f5a1f024ee766c62392240b69a9e1a11ab9e Sep 18 17:04:45.084794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:45.084758 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-v8nnn" event={"ID":"10b057e4-8030-4311-a307-fb2f16b3d35a","Type":"ContainerStarted","Data":"b7aa1b80896528f11b92230afeadb30e900c98c33a6aef876f67f829546a3880"} Sep 18 17:04:45.086228 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:45.086174 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-gpwpm" event={"ID":"88eaa60c-05b5-4f53-8212-2e86a4b4e897","Type":"ContainerStarted","Data":"07d88f4ffd8f74d66746d24e3790f5a1f024ee766c62392240b69a9e1a11ab9e"} Sep 18 17:04:49.109585 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:49.109549 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-v8nnn" event={"ID":"10b057e4-8030-4311-a307-fb2f16b3d35a","Type":"ContainerStarted","Data":"f999c7109243dac61f0a2f42220f6bd0f9f344b88c94c6a4ef2c116be231d37a"} Sep 18 17:04:49.110045 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:49.109680 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:04:49.111145 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:49.111123 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-gpwpm" event={"ID":"88eaa60c-05b5-4f53-8212-2e86a4b4e897","Type":"ContainerStarted","Data":"64545d67a8ac06fe2020bed60a5223a43744f3f1071e8976b41a3f89fb7abb37"} Sep 18 17:04:49.111267 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:49.111253 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:04:49.138253 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:49.138207 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-v8nnn" podStartSLOduration=2.531512052 podStartE2EDuration="6.138191975s" totalImagesPullingTime="3.606679923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:04:49.136891719 +0000 UTC m=+662.471256426" watchObservedRunningTime="2026-09-18 17:04:49.138191975 +0000 UTC m=+662.472556683" Sep 18 17:04:49.158600 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:04:49.158530 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-gpwpm" podStartSLOduration=2.52520936 podStartE2EDuration="6.158516133s" totalImagesPullingTime="3.633306773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:04:49.158178782 +0000 UTC m=+662.492543489" watchObservedRunningTime="2026-09-18 17:04:49.158516133 +0000 UTC m=+662.492880839" Sep 18 17:05:00.119887 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:05:00.119854 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-gpwpm" Sep 18 17:05:00.122323 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:05:00.122301 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-v8nnn" Sep 18 17:08:46.805951 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:08:46.805845 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:08:46.809154 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:08:46.806666 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:11:21.163758 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.163720 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl"] Sep 18 17:11:21.172527 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.172499 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.175045 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.175021 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 18 17:11:21.175629 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.175608 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\"" Sep 18 17:11:21.175757 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.175620 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\"" Sep 18 17:11:21.175757 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.175627 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\"" Sep 18 17:11:21.175757 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.175690 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 18 17:11:21.176903 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.176881 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl"] Sep 18 17:11:21.298517 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.298486 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-dxbbl\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.298517 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.298518 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a23bedd-7b5b-46e9-be14-7089aef48e88-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-dxbbl\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.298737 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.298537 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnx6\" (UniqueName: \"kubernetes.io/projected/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-api-access-zlnx6\") pod \"message-dumper-predictor-6575bcfdfd-dxbbl\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.399331 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.399296 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-dxbbl\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.399331 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.399331 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a23bedd-7b5b-46e9-be14-7089aef48e88-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-dxbbl\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.399602 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.399352 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnx6\" (UniqueName: \"kubernetes.io/projected/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-api-access-zlnx6\") pod \"message-dumper-predictor-6575bcfdfd-dxbbl\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.399974 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.399950 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-dxbbl\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.401904 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.401882 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a23bedd-7b5b-46e9-be14-7089aef48e88-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-dxbbl\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.409016 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.408991 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnx6\" (UniqueName: \"kubernetes.io/projected/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-api-access-zlnx6\") pod \"message-dumper-predictor-6575bcfdfd-dxbbl\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.485040 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.484973 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:21.760211 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.760180 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl"] Sep 18 17:11:21.762732 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:11:21.762702 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a23bedd_7b5b_46e9_be14_7089aef48e88.slice/crio-29adbc99e6e5a8eb5f1b64e0cb75ee5cef398a2f985b1c6bc203f3d07dd6441a WatchSource:0}: Error finding container 29adbc99e6e5a8eb5f1b64e0cb75ee5cef398a2f985b1c6bc203f3d07dd6441a: Status 404 returned error can't find the container with id 29adbc99e6e5a8eb5f1b64e0cb75ee5cef398a2f985b1c6bc203f3d07dd6441a Sep 18 17:11:21.764685 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.764666 2874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:11:21.872806 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:21.872766 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" event={"ID":"1a23bedd-7b5b-46e9-be14-7089aef48e88","Type":"ContainerStarted","Data":"29adbc99e6e5a8eb5f1b64e0cb75ee5cef398a2f985b1c6bc203f3d07dd6441a"} Sep 18 17:11:23.886523 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:23.886483 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" event={"ID":"1a23bedd-7b5b-46e9-be14-7089aef48e88","Type":"ContainerStarted","Data":"dbf9112c48f7a06d9ce8d0bff56ed6cab70dcc481f7814c8b6f435e46f306ab8"} Sep 18 17:11:25.899725 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:25.899688 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" event={"ID":"1a23bedd-7b5b-46e9-be14-7089aef48e88","Type":"ContainerStarted","Data":"5ea753b4724f6b74e598114482841e92483b5245f049171d60ba48e6b654cac6"} Sep 18 17:11:25.900163 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:25.899783 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:25.921944 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:25.921891 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" podStartSLOduration=1.4563638 podStartE2EDuration="4.921873231s" totalImagesPullingTime="3.465509431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:11:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:11:25.92013097 +0000 UTC m=+1059.254495677" watchObservedRunningTime="2026-09-18 17:11:25.921873231 +0000 UTC m=+1059.256237937" Sep 18 17:11:26.904953 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:26.904919 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:26.906974 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:26.906952 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:11:33.918835 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:11:33.918803 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:13:26.800986 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:26.800953 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-dxbbl_1a23bedd-7b5b-46e9-be14-7089aef48e88/kserve-container/0.log" Sep 18 17:13:27.243087 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.243054 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl"] Sep 18 17:13:27.243283 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.243261 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" podUID="1a23bedd-7b5b-46e9-be14-7089aef48e88" containerName="kserve-container" containerID="cri-o://dbf9112c48f7a06d9ce8d0bff56ed6cab70dcc481f7814c8b6f435e46f306ab8" gracePeriod=30 Sep 18 17:13:27.243378 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.243348 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" podUID="1a23bedd-7b5b-46e9-be14-7089aef48e88" containerName="kube-rbac-proxy" containerID="cri-o://5ea753b4724f6b74e598114482841e92483b5245f049171d60ba48e6b654cac6" gracePeriod=30 Sep 18 17:13:27.446325 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.446285 2874 generic.go:356] "Generic (PLEG): container finished" podID="1a23bedd-7b5b-46e9-be14-7089aef48e88" containerID="5ea753b4724f6b74e598114482841e92483b5245f049171d60ba48e6b654cac6" exitCode=2 Sep 18 17:13:27.446325 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.446319 2874 generic.go:356] "Generic (PLEG): container finished" podID="1a23bedd-7b5b-46e9-be14-7089aef48e88" containerID="dbf9112c48f7a06d9ce8d0bff56ed6cab70dcc481f7814c8b6f435e46f306ab8" exitCode=2 Sep 18 17:13:27.446604 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.446353 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" event={"ID":"1a23bedd-7b5b-46e9-be14-7089aef48e88","Type":"ContainerDied","Data":"5ea753b4724f6b74e598114482841e92483b5245f049171d60ba48e6b654cac6"} Sep 18 17:13:27.446604 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.446435 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" event={"ID":"1a23bedd-7b5b-46e9-be14-7089aef48e88","Type":"ContainerDied","Data":"dbf9112c48f7a06d9ce8d0bff56ed6cab70dcc481f7814c8b6f435e46f306ab8"} Sep 18 17:13:27.482028 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.482002 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:13:27.537066 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.537002 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlnx6\" (UniqueName: \"kubernetes.io/projected/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-api-access-zlnx6\") pod \"1a23bedd-7b5b-46e9-be14-7089aef48e88\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " Sep 18 17:13:27.537200 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.537071 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a23bedd-7b5b-46e9-be14-7089aef48e88-proxy-tls\") pod \"1a23bedd-7b5b-46e9-be14-7089aef48e88\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " Sep 18 17:13:27.537200 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.537150 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-rbac-proxy-sar-config\") pod \"1a23bedd-7b5b-46e9-be14-7089aef48e88\" (UID: \"1a23bedd-7b5b-46e9-be14-7089aef48e88\") " Sep 18 17:13:27.537575 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.537546 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a23bedd-7b5b-46e9-be14-7089aef48e88" (UID: "1a23bedd-7b5b-46e9-be14-7089aef48e88"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:13:27.539240 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.539219 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-api-access-zlnx6" (OuterVolumeSpecName: "kube-api-access-zlnx6") pod "1a23bedd-7b5b-46e9-be14-7089aef48e88" (UID: "1a23bedd-7b5b-46e9-be14-7089aef48e88"). InnerVolumeSpecName "kube-api-access-zlnx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:13:27.539420 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.539247 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a23bedd-7b5b-46e9-be14-7089aef48e88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a23bedd-7b5b-46e9-be14-7089aef48e88" (UID: "1a23bedd-7b5b-46e9-be14-7089aef48e88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:13:27.638123 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.638092 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a23bedd-7b5b-46e9-be14-7089aef48e88-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:13:27.638123 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.638118 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:13:27.638123 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:27.638128 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlnx6\" (UniqueName: \"kubernetes.io/projected/1a23bedd-7b5b-46e9-be14-7089aef48e88-kube-api-access-zlnx6\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:13:28.453132 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:28.453095 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" event={"ID":"1a23bedd-7b5b-46e9-be14-7089aef48e88","Type":"ContainerDied","Data":"29adbc99e6e5a8eb5f1b64e0cb75ee5cef398a2f985b1c6bc203f3d07dd6441a"} Sep 18 17:13:28.453651 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:28.453137 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl" Sep 18 17:13:28.453651 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:28.453161 2874 scope.go:117] "RemoveContainer" containerID="5ea753b4724f6b74e598114482841e92483b5245f049171d60ba48e6b654cac6" Sep 18 17:13:28.462781 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:28.462765 2874 scope.go:117] "RemoveContainer" containerID="dbf9112c48f7a06d9ce8d0bff56ed6cab70dcc481f7814c8b6f435e46f306ab8" Sep 18 17:13:28.475597 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:28.475574 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl"] Sep 18 17:13:28.479511 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:28.479491 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-dxbbl"] Sep 18 17:13:28.748880 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:28.748805 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a23bedd-7b5b-46e9-be14-7089aef48e88" path="/var/lib/kubelet/pods/1a23bedd-7b5b-46e9-be14-7089aef48e88/volumes" Sep 18 17:13:42.345180 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.345141 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf"] Sep 18 17:13:42.345610 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.345543 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a23bedd-7b5b-46e9-be14-7089aef48e88" containerName="kserve-container" Sep 18 17:13:42.345610 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.345557 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a23bedd-7b5b-46e9-be14-7089aef48e88" containerName="kserve-container" Sep 18 17:13:42.345610 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.345572 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a23bedd-7b5b-46e9-be14-7089aef48e88" containerName="kube-rbac-proxy" Sep 18 17:13:42.345610 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.345577 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a23bedd-7b5b-46e9-be14-7089aef48e88" containerName="kube-rbac-proxy" Sep 18 17:13:42.345766 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.345634 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a23bedd-7b5b-46e9-be14-7089aef48e88" containerName="kserve-container" Sep 18 17:13:42.345766 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.345645 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a23bedd-7b5b-46e9-be14-7089aef48e88" containerName="kube-rbac-proxy" Sep 18 17:13:42.350281 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.350259 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.353004 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.352982 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\"" Sep 18 17:13:42.353004 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.352995 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\"" Sep 18 17:13:42.353157 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.352997 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 18 17:13:42.354213 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.354190 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\"" Sep 18 17:13:42.354213 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.354206 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 18 17:13:42.354360 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.354208 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 18 17:13:42.354360 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.354238 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 18 17:13:42.362606 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.362583 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf"] Sep 18 17:13:42.460190 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.460157 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.460190 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.460189 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.460394 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.460219 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnkmd\" (UniqueName: \"kubernetes.io/projected/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-api-access-wnkmd\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.460394 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.460284 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.460394 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.460340 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.561563 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.561524 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.561770 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.561581 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.561770 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.561616 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.561770 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.561663 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnkmd\" (UniqueName: \"kubernetes.io/projected/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-api-access-wnkmd\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.561770 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.561701 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.562019 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.561988 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.562481 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.562456 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.562633 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.562614 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.564562 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.564544 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.571237 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.571181 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnkmd\" (UniqueName: \"kubernetes.io/projected/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-api-access-wnkmd\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.661242 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.661210 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:13:42.936919 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:42.936832 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf"] Sep 18 17:13:42.940042 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:13:42.940016 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e10dd3_0b1a_4e2b_a2b6_fcea0bee7f3c.slice/crio-05f76bee690745e7f808d61f74a0060aa0473784cdf058fa15c5e403075d73a9 WatchSource:0}: Error finding container 05f76bee690745e7f808d61f74a0060aa0473784cdf058fa15c5e403075d73a9: Status 404 returned error can't find the container with id 05f76bee690745e7f808d61f74a0060aa0473784cdf058fa15c5e403075d73a9 Sep 18 17:13:43.522701 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:43.522666 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" event={"ID":"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c","Type":"ContainerStarted","Data":"05f76bee690745e7f808d61f74a0060aa0473784cdf058fa15c5e403075d73a9"} Sep 18 17:13:46.835647 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:46.835544 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:13:46.835647 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:46.835545 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:13:47.546448 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:47.546414 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" event={"ID":"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c","Type":"ContainerStarted","Data":"72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6"} Sep 18 17:13:50.566336 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:50.566306 2874 generic.go:356] "Generic (PLEG): container finished" podID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerID="72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6" exitCode=0 Sep 18 17:13:50.566708 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:13:50.566373 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" event={"ID":"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c","Type":"ContainerDied","Data":"72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6"} Sep 18 17:14:15.717833 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:14:15.717797 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" event={"ID":"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c","Type":"ContainerStarted","Data":"bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95"} Sep 18 17:14:15.718266 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:14:15.717866 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" event={"ID":"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c","Type":"ContainerStarted","Data":"0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb"} Sep 18 17:14:15.718266 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:14:15.718079 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:14:15.754475 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:14:15.754422 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" podStartSLOduration=1.6120626850000002 podStartE2EDuration="33.754387981s" totalImagesPullingTime="28.266522662s" totalInitContainerRuntime="3.875802634s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:13:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:14:15.753427526 +0000 UTC m=+1229.087792235" watchObservedRunningTime="2026-09-18 17:14:15.754387981 +0000 UTC m=+1229.088752687" Sep 18 17:14:16.722287 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:14:16.722258 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:14:22.732870 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:14:22.732838 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:14:52.734861 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:14:52.734824 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 17:15:02.733696 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:02.733644 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 17:15:12.733420 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:12.733359 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 17:15:22.734215 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:22.734128 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 17:15:32.733699 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:32.733656 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 17:15:43.331744 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:43.331703 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:15:52.487295 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:52.487260 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf"] Sep 18 17:15:52.487774 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:52.487526 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kserve-container" containerID="cri-o://0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb" gracePeriod=30 Sep 18 17:15:52.487774 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:52.487608 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kube-rbac-proxy" containerID="cri-o://bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95" gracePeriod=30 Sep 18 17:15:52.727577 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:52.727533 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused" Sep 18 17:15:52.734211 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:52.734180 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused" Sep 18 17:15:53.164024 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:53.163991 2874 generic.go:356] "Generic (PLEG): container finished" podID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerID="bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95" exitCode=2 Sep 18 17:15:53.164198 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:53.164049 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" event={"ID":"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c","Type":"ContainerDied","Data":"bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95"} Sep 18 17:15:57.639230 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.639204 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:15:57.740983 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.740905 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj"] Sep 18 17:15:57.741241 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.741229 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kserve-container" Sep 18 17:15:57.741292 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.741243 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kserve-container" Sep 18 17:15:57.741292 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.741266 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="storage-initializer" Sep 18 17:15:57.741292 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.741271 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="storage-initializer" Sep 18 17:15:57.741292 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.741282 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kube-rbac-proxy" Sep 18 17:15:57.741292 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.741287 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kube-rbac-proxy" Sep 18 17:15:57.741486 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.741340 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kube-rbac-proxy" Sep 18 17:15:57.741486 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.741350 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerName="kserve-container" Sep 18 17:15:57.744212 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.744191 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:57.747932 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.747910 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\"" Sep 18 17:15:57.748078 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.748061 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\"" Sep 18 17:15:57.759496 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.759473 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj"] Sep 18 17:15:57.811556 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.811526 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-cabundle-cert\") pod \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " Sep 18 17:15:57.811736 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.811564 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnkmd\" (UniqueName: \"kubernetes.io/projected/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-api-access-wnkmd\") pod \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " Sep 18 17:15:57.811736 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.811593 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-proxy-tls\") pod \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " Sep 18 17:15:57.811736 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.811618 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kserve-provision-location\") pod \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " Sep 18 17:15:57.811736 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.811660 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-rbac-proxy-sar-config\") pod \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\" (UID: \"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c\") " Sep 18 17:15:57.811979 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.811951 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" (UID: "a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:15:57.812040 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.812011 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" (UID: "a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:15:57.812082 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.812064 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" (UID: "a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:15:57.813842 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.813819 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" (UID: "a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:15:57.813937 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.813920 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-api-access-wnkmd" (OuterVolumeSpecName: "kube-api-access-wnkmd") pod "a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" (UID: "a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c"). InnerVolumeSpecName "kube-api-access-wnkmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:15:57.912605 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.912572 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:57.912788 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.912765 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:57.912858 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.912836 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8gc\" (UniqueName: \"kubernetes.io/projected/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-api-access-qx8gc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:57.912908 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.912883 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a2856f2-15f3-4237-a1d8-34a199462b37-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:57.912946 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.912934 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:57.913003 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.912991 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:15:57.913041 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.913008 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:15:57.913041 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.913016 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:15:57.913041 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.913025 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:15:57.913041 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:57.913034 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnkmd\" (UniqueName: \"kubernetes.io/projected/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c-kube-api-access-wnkmd\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:15:58.013826 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.013747 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:58.013826 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.013782 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8gc\" (UniqueName: \"kubernetes.io/projected/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-api-access-qx8gc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:58.013826 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.013804 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a2856f2-15f3-4237-a1d8-34a199462b37-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:58.014079 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.013832 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:58.014079 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.013880 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:58.014079 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:15:58.013999 2874 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found Sep 18 17:15:58.014079 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:15:58.014078 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls podName:4a2856f2-15f3-4237-a1d8-34a199462b37 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:58.514056741 +0000 UTC m=+1331.848421432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" (UID: "4a2856f2-15f3-4237-a1d8-34a199462b37") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found Sep 18 17:15:58.014291 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.014243 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a2856f2-15f3-4237-a1d8-34a199462b37-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:58.014478 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.014457 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:58.014645 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.014626 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:58.023857 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.023838 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8gc\" (UniqueName: \"kubernetes.io/projected/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-api-access-qx8gc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:58.189204 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.189173 2874 generic.go:356] "Generic (PLEG): container finished" podID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" containerID="0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb" exitCode=0 Sep 18 17:15:58.189374 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.189231 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" Sep 18 17:15:58.189374 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.189258 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" event={"ID":"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c","Type":"ContainerDied","Data":"0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb"} Sep 18 17:15:58.189374 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.189330 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf" event={"ID":"a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c","Type":"ContainerDied","Data":"05f76bee690745e7f808d61f74a0060aa0473784cdf058fa15c5e403075d73a9"} Sep 18 17:15:58.189374 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.189359 2874 scope.go:117] "RemoveContainer" containerID="bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95" Sep 18 17:15:58.198529 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.198513 2874 scope.go:117] "RemoveContainer" containerID="0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb" Sep 18 17:15:58.208594 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.208578 2874 scope.go:117] "RemoveContainer" containerID="72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6" Sep 18 17:15:58.215210 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.215186 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf"] Sep 18 17:15:58.217648 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.217617 2874 scope.go:117] "RemoveContainer" containerID="bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95" Sep 18 17:15:58.218467 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:15:58.217993 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95\": container with ID starting with bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95 not found: ID does not exist" containerID="bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95" Sep 18 17:15:58.218467 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.218024 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95"} err="failed to get container status \"bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95\": rpc error: code = NotFound desc = could not find container \"bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95\": container with ID starting with bc5e65c4e37a3fcca426d189656c5539f69ca479c103f02395e72c35b9994a95 not found: ID does not exist" Sep 18 17:15:58.218467 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.218048 2874 scope.go:117] "RemoveContainer" containerID="0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb" Sep 18 17:15:58.218643 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:15:58.218509 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb\": container with ID starting with 0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb not found: ID does not exist" containerID="0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb" Sep 18 17:15:58.218643 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.218540 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb"} err="failed to get container status \"0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb\": rpc error: code = NotFound desc = could not find container \"0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb\": container with ID starting with 0a19bf2a0ed33860cc6426ec9244ed09bc5d3f2112188b5f8fb26fe494ad50bb not found: ID does not exist" Sep 18 17:15:58.218643 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.218568 2874 scope.go:117] "RemoveContainer" containerID="72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6" Sep 18 17:15:58.218846 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:15:58.218826 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6\": container with ID starting with 72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6 not found: ID does not exist" containerID="72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6" Sep 18 17:15:58.218888 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.218854 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6"} err="failed to get container status \"72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6\": rpc error: code = NotFound desc = could not find container \"72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6\": container with ID starting with 72505e40900617b6622367344d67dce80b2b4fbb7bc369c9b163f2e5009a23d6 not found: ID does not exist" Sep 18 17:15:58.219839 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.219821 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-zkspf"] Sep 18 17:15:58.517613 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.517587 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:58.517786 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:15:58.517733 2874 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found Sep 18 17:15:58.517830 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:15:58.517792 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls podName:4a2856f2-15f3-4237-a1d8-34a199462b37 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:59.517775983 +0000 UTC m=+1332.852140853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" (UID: "4a2856f2-15f3-4237-a1d8-34a199462b37") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found Sep 18 17:15:58.749333 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:58.749298 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c" path="/var/lib/kubelet/pods/a0e10dd3-0b1a-4e2b-a2b6-fcea0bee7f3c/volumes" Sep 18 17:15:59.525965 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:59.525934 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:59.528447 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:59.528428 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:59.554684 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:59.554659 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:15:59.838104 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:15:59.838080 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj"] Sep 18 17:15:59.839937 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:15:59.839911 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2856f2_15f3_4237_a1d8_34a199462b37.slice/crio-b9483abfab7f7a01f9ab0130971dcb63244bdcb91d1a79ecabee9ffed12f76de WatchSource:0}: Error finding container b9483abfab7f7a01f9ab0130971dcb63244bdcb91d1a79ecabee9ffed12f76de: Status 404 returned error can't find the container with id b9483abfab7f7a01f9ab0130971dcb63244bdcb91d1a79ecabee9ffed12f76de Sep 18 17:16:00.202239 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:00.202201 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" event={"ID":"4a2856f2-15f3-4237-a1d8-34a199462b37","Type":"ContainerStarted","Data":"7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07"} Sep 18 17:16:00.202450 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:00.202274 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" event={"ID":"4a2856f2-15f3-4237-a1d8-34a199462b37","Type":"ContainerStarted","Data":"b9483abfab7f7a01f9ab0130971dcb63244bdcb91d1a79ecabee9ffed12f76de"} Sep 18 17:16:04.221054 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:04.221017 2874 generic.go:356] "Generic (PLEG): container finished" podID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerID="7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07" exitCode=0 Sep 18 17:16:04.221440 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:04.221088 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" event={"ID":"4a2856f2-15f3-4237-a1d8-34a199462b37","Type":"ContainerDied","Data":"7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07"} Sep 18 17:16:05.228769 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:05.228739 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" event={"ID":"4a2856f2-15f3-4237-a1d8-34a199462b37","Type":"ContainerStarted","Data":"a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59"} Sep 18 17:16:05.229168 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:05.228803 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" event={"ID":"4a2856f2-15f3-4237-a1d8-34a199462b37","Type":"ContainerStarted","Data":"249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2"} Sep 18 17:16:05.229168 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:05.229000 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:16:05.252285 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:05.252234 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" podStartSLOduration=4.517058438 podStartE2EDuration="8.252218922s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.735160484s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:05.251314186 +0000 UTC m=+1338.585678884" watchObservedRunningTime="2026-09-18 17:16:05.252218922 +0000 UTC m=+1338.586583626" Sep 18 17:16:06.234285 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:06.234245 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:16:12.244411 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:12.244368 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:16:29.064671 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.064639 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz"] Sep 18 17:16:29.068319 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.068299 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.070811 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.070788 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\"" Sep 18 17:16:29.070921 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.070792 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\"" Sep 18 17:16:29.080978 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.080957 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz"] Sep 18 17:16:29.171244 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.171197 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.171244 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.171241 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.171533 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.171269 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.171533 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.171349 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.171533 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.171427 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k4d5\" (UniqueName: \"kubernetes.io/projected/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-api-access-6k4d5\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.272775 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.272731 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.272988 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.272792 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4d5\" (UniqueName: \"kubernetes.io/projected/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-api-access-6k4d5\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.272988 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.272835 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.272988 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.272856 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.272988 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.272882 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.273297 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.273276 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.273584 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.273562 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.273665 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.273642 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.275503 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.275485 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.282818 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.282788 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4d5\" (UniqueName: \"kubernetes.io/projected/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-api-access-6k4d5\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-2szzz\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.378954 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.378930 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:16:29.653077 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:29.652858 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz"] Sep 18 17:16:29.655567 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:16:29.655532 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d50a42_c5b2_4bfb_8b71_6fd55efef75d.slice/crio-a58b5311e7c8b9858b3d10521767a6443720272609b9d0e1d5f0a9a379f912e3 WatchSource:0}: Error finding container a58b5311e7c8b9858b3d10521767a6443720272609b9d0e1d5f0a9a379f912e3: Status 404 returned error can't find the container with id a58b5311e7c8b9858b3d10521767a6443720272609b9d0e1d5f0a9a379f912e3 Sep 18 17:16:30.354519 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:30.354483 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" event={"ID":"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d","Type":"ContainerStarted","Data":"23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb"} Sep 18 17:16:30.354975 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:30.354538 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" event={"ID":"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d","Type":"ContainerStarted","Data":"a58b5311e7c8b9858b3d10521767a6443720272609b9d0e1d5f0a9a379f912e3"} Sep 18 17:16:37.389084 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:37.389049 2874 generic.go:356] "Generic (PLEG): container finished" podID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerID="23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb" exitCode=0 Sep 18 17:16:37.389483 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:37.389124 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" event={"ID":"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d","Type":"ContainerDied","Data":"23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb"} Sep 18 17:16:37.390546 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:37.390528 2874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:16:42.245517 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:42.245475 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 17:16:52.245204 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:16:52.245165 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 17:17:02.245194 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:02.245157 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 17:17:12.245473 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:12.245381 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 17:17:22.244953 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:22.244915 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 17:17:33.032872 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:33.032836 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:17:37.745053 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:37.745017 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj"] Sep 18 17:17:37.745470 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:37.745274 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kserve-container" containerID="cri-o://249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2" gracePeriod=30 Sep 18 17:17:37.745470 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:37.745356 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kube-rbac-proxy" containerID="cri-o://a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59" gracePeriod=30 Sep 18 17:17:38.733757 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:38.733719 2874 generic.go:356] "Generic (PLEG): container finished" podID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerID="a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59" exitCode=2 Sep 18 17:17:38.733939 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:38.733762 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" event={"ID":"4a2856f2-15f3-4237-a1d8-34a199462b37","Type":"ContainerDied","Data":"a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59"} Sep 18 17:17:42.239593 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:42.239554 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Sep 18 17:17:42.244974 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:42.244947 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused" Sep 18 17:17:43.360804 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.360778 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:17:43.439529 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.439459 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx8gc\" (UniqueName: \"kubernetes.io/projected/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-api-access-qx8gc\") pod \"4a2856f2-15f3-4237-a1d8-34a199462b37\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " Sep 18 17:17:43.439529 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.439511 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-cabundle-cert\") pod \"4a2856f2-15f3-4237-a1d8-34a199462b37\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " Sep 18 17:17:43.439758 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.439551 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a2856f2-15f3-4237-a1d8-34a199462b37-kserve-provision-location\") pod \"4a2856f2-15f3-4237-a1d8-34a199462b37\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " Sep 18 17:17:43.439758 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.439573 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-rbac-proxy-sar-config\") pod \"4a2856f2-15f3-4237-a1d8-34a199462b37\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " Sep 18 17:17:43.439758 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.439654 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls\") pod \"4a2856f2-15f3-4237-a1d8-34a199462b37\" (UID: \"4a2856f2-15f3-4237-a1d8-34a199462b37\") " Sep 18 17:17:43.439967 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.439936 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2856f2-15f3-4237-a1d8-34a199462b37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a2856f2-15f3-4237-a1d8-34a199462b37" (UID: "4a2856f2-15f3-4237-a1d8-34a199462b37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:17:43.439967 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.439944 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a2856f2-15f3-4237-a1d8-34a199462b37" (UID: "4a2856f2-15f3-4237-a1d8-34a199462b37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:17:43.439967 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.439962 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a2856f2-15f3-4237-a1d8-34a199462b37" (UID: "4a2856f2-15f3-4237-a1d8-34a199462b37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:17:43.442168 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.442141 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a2856f2-15f3-4237-a1d8-34a199462b37" (UID: "4a2856f2-15f3-4237-a1d8-34a199462b37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:17:43.442255 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.442219 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-api-access-qx8gc" (OuterVolumeSpecName: "kube-api-access-qx8gc") pod "4a2856f2-15f3-4237-a1d8-34a199462b37" (UID: "4a2856f2-15f3-4237-a1d8-34a199462b37"). InnerVolumeSpecName "kube-api-access-qx8gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:17:43.540714 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.540678 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a2856f2-15f3-4237-a1d8-34a199462b37-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:17:43.540714 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.540710 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:17:43.540916 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.540724 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a2856f2-15f3-4237-a1d8-34a199462b37-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:17:43.540916 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.540739 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8gc\" (UniqueName: \"kubernetes.io/projected/4a2856f2-15f3-4237-a1d8-34a199462b37-kube-api-access-qx8gc\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:17:43.540916 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.540750 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a2856f2-15f3-4237-a1d8-34a199462b37-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:17:43.763595 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.763500 2874 generic.go:356] "Generic (PLEG): container finished" podID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerID="249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2" exitCode=0 Sep 18 17:17:43.763595 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.763575 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" event={"ID":"4a2856f2-15f3-4237-a1d8-34a199462b37","Type":"ContainerDied","Data":"249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2"} Sep 18 17:17:43.763595 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.763585 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" Sep 18 17:17:43.763846 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.763628 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj" event={"ID":"4a2856f2-15f3-4237-a1d8-34a199462b37","Type":"ContainerDied","Data":"b9483abfab7f7a01f9ab0130971dcb63244bdcb91d1a79ecabee9ffed12f76de"} Sep 18 17:17:43.763846 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.763668 2874 scope.go:117] "RemoveContainer" containerID="a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59" Sep 18 17:17:43.774309 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.774288 2874 scope.go:117] "RemoveContainer" containerID="249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2" Sep 18 17:17:43.783966 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.783945 2874 scope.go:117] "RemoveContainer" containerID="7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07" Sep 18 17:17:43.793813 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.793794 2874 scope.go:117] "RemoveContainer" containerID="a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59" Sep 18 17:17:43.794126 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:17:43.794105 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59\": container with ID starting with a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59 not found: ID does not exist" containerID="a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59" Sep 18 17:17:43.794214 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.794135 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59"} err="failed to get container status \"a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59\": rpc error: code = NotFound desc = could not find container \"a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59\": container with ID starting with a9baa450786cf6c8364a076a411bb7e5748c55f0cd5c39cb748cac958ba94c59 not found: ID does not exist" Sep 18 17:17:43.794214 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.794159 2874 scope.go:117] "RemoveContainer" containerID="249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2" Sep 18 17:17:43.794486 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:17:43.794461 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2\": container with ID starting with 249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2 not found: ID does not exist" containerID="249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2" Sep 18 17:17:43.794548 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.794495 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2"} err="failed to get container status \"249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2\": rpc error: code = NotFound desc = could not find container \"249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2\": container with ID starting with 249f0cd4c215ec0ba8e325b6862ed11d4c1ba541d333736eac771ed9ca1d84b2 not found: ID does not exist" Sep 18 17:17:43.794548 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.794517 2874 scope.go:117] "RemoveContainer" containerID="7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07" Sep 18 17:17:43.794839 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:17:43.794822 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07\": container with ID starting with 7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07 not found: ID does not exist" containerID="7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07" Sep 18 17:17:43.794900 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.794853 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07"} err="failed to get container status \"7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07\": rpc error: code = NotFound desc = could not find container \"7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07\": container with ID starting with 7ae3ec9d7b394d49b13d87feb1b96b7dab49c703719ecabb3109cdabc6109d07 not found: ID does not exist" Sep 18 17:17:43.809809 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.809781 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj"] Sep 18 17:17:43.813503 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:43.813480 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gt9mj"] Sep 18 17:17:44.750239 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:17:44.750206 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" path="/var/lib/kubelet/pods/4a2856f2-15f3-4237-a1d8-34a199462b37/volumes" Sep 18 17:18:38.052642 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:18:38.052598 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" event={"ID":"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d","Type":"ContainerStarted","Data":"8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a"} Sep 18 17:18:38.053040 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:18:38.052668 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" event={"ID":"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d","Type":"ContainerStarted","Data":"cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61"} Sep 18 17:18:38.053040 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:18:38.052764 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:18:38.076104 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:18:38.076054 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" podStartSLOduration=2.465152081 podStartE2EDuration="2m9.076041545s" totalImagesPullingTime="1m59.66079285s" totalInitContainerRuntime="6.950096614s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:18:38.074761881 +0000 UTC m=+1491.409126612" watchObservedRunningTime="2026-09-18 17:18:38.076041545 +0000 UTC m=+1491.410406252" Sep 18 17:18:39.057753 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:18:39.057724 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:18:45.067811 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:18:45.067777 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:18:46.869337 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:18:46.869242 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:18:47.081874 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:18:46.871010 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:20:12.776283 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:12.776252 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:20:23.785624 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:23.785575 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 18 17:20:25.368850 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:25.368815 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz"] Sep 18 17:20:25.369464 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:25.369077 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="kserve-container" containerID="cri-o://cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61" gracePeriod=30 Sep 18 17:20:25.369464 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:25.369154 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="kube-rbac-proxy" containerID="cri-o://8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a" gracePeriod=30 Sep 18 17:20:25.541736 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:25.541705 2874 generic.go:356] "Generic (PLEG): container finished" podID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerID="8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a" exitCode=2 Sep 18 17:20:25.541903 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:25.541743 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" event={"ID":"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d","Type":"ContainerDied","Data":"8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a"} Sep 18 17:20:26.817918 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.817891 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:20:26.906803 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.906776 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-rbac-proxy-sar-config\") pod \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " Sep 18 17:20:26.906957 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.906840 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-proxy-tls\") pod \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " Sep 18 17:20:26.906957 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.906940 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k4d5\" (UniqueName: \"kubernetes.io/projected/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-api-access-6k4d5\") pod \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " Sep 18 17:20:26.907052 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.907033 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kserve-provision-location\") pod \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " Sep 18 17:20:26.907094 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.907078 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-cabundle-cert\") pod \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\" (UID: \"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d\") " Sep 18 17:20:26.907216 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.907194 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" (UID: "b8d50a42-c5b2-4bfb-8b71-6fd55efef75d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:20:26.907580 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.907551 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" (UID: "b8d50a42-c5b2-4bfb-8b71-6fd55efef75d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:20:26.909220 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.909195 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-api-access-6k4d5" (OuterVolumeSpecName: "kube-api-access-6k4d5") pod "b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" (UID: "b8d50a42-c5b2-4bfb-8b71-6fd55efef75d"). InnerVolumeSpecName "kube-api-access-6k4d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:20:26.909220 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.909205 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" (UID: "b8d50a42-c5b2-4bfb-8b71-6fd55efef75d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:20:26.912768 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:26.912738 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" (UID: "b8d50a42-c5b2-4bfb-8b71-6fd55efef75d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:20:27.008247 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.008219 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k4d5\" (UniqueName: \"kubernetes.io/projected/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-api-access-6k4d5\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:20:27.008247 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.008242 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:20:27.008247 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.008252 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:20:27.008489 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.008261 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:20:27.008489 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.008269 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:20:27.552938 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.552897 2874 generic.go:356] "Generic (PLEG): container finished" podID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerID="cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61" exitCode=0 Sep 18 17:20:27.553108 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.552954 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" Sep 18 17:20:27.553108 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.552978 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" event={"ID":"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d","Type":"ContainerDied","Data":"cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61"} Sep 18 17:20:27.553108 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.553036 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz" event={"ID":"b8d50a42-c5b2-4bfb-8b71-6fd55efef75d","Type":"ContainerDied","Data":"a58b5311e7c8b9858b3d10521767a6443720272609b9d0e1d5f0a9a379f912e3"} Sep 18 17:20:27.553108 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.553064 2874 scope.go:117] "RemoveContainer" containerID="8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a" Sep 18 17:20:27.562545 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.562527 2874 scope.go:117] "RemoveContainer" containerID="cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61" Sep 18 17:20:27.570224 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.570209 2874 scope.go:117] "RemoveContainer" containerID="23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb" Sep 18 17:20:27.577964 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.577940 2874 scope.go:117] "RemoveContainer" containerID="8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a" Sep 18 17:20:27.578334 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:20:27.578313 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a\": container with ID starting with 8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a not found: ID does not exist" containerID="8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a" Sep 18 17:20:27.578522 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.578345 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a"} err="failed to get container status \"8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a\": rpc error: code = NotFound desc = could not find container \"8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a\": container with ID starting with 8b2c305113ea4607ef39e46ffdd964d20028763503a9ec9d00added900e9386a not found: ID does not exist" Sep 18 17:20:27.578522 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.578368 2874 scope.go:117] "RemoveContainer" containerID="cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61" Sep 18 17:20:27.578720 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:20:27.578703 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61\": container with ID starting with cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61 not found: ID does not exist" containerID="cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61" Sep 18 17:20:27.578765 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.578728 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61"} err="failed to get container status \"cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61\": rpc error: code = NotFound desc = could not find container \"cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61\": container with ID starting with cf0f67f197c35cee6db61a99f2bf885a24debe9ee68c44dc003d1287c3b8fb61 not found: ID does not exist" Sep 18 17:20:27.578765 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.578743 2874 scope.go:117] "RemoveContainer" containerID="23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb" Sep 18 17:20:27.579010 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:20:27.578988 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb\": container with ID starting with 23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb not found: ID does not exist" containerID="23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb" Sep 18 17:20:27.579069 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.579019 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb"} err="failed to get container status \"23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb\": rpc error: code = NotFound desc = could not find container \"23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb\": container with ID starting with 23721d55c056f63bea2cd4e77b582f55089b445b269e92fa7f7bde4e01a168bb not found: ID does not exist" Sep 18 17:20:27.580157 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.580139 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz"] Sep 18 17:20:27.583366 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:27.583346 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-2szzz"] Sep 18 17:20:28.749228 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:28.749183 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" path="/var/lib/kubelet/pods/b8d50a42-c5b2-4bfb-8b71-6fd55efef75d/volumes" Sep 18 17:20:30.748367 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748333 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6"] Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748661 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="kube-rbac-proxy" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748672 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="kube-rbac-proxy" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748688 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="storage-initializer" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748694 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="storage-initializer" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748700 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="kserve-container" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748705 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="kserve-container" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748714 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kube-rbac-proxy" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748719 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kube-rbac-proxy" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748726 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="storage-initializer" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748730 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="storage-initializer" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748742 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kserve-container" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748746 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kserve-container" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748795 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kube-rbac-proxy" Sep 18 17:20:30.748794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748803 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="kube-rbac-proxy" Sep 18 17:20:30.749298 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748811 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2856f2-15f3-4237-a1d8-34a199462b37" containerName="kserve-container" Sep 18 17:20:30.749298 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.748817 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d50a42-c5b2-4bfb-8b71-6fd55efef75d" containerName="kserve-container" Sep 18 17:20:30.752503 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.752484 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.759049 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.758713 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 18 17:20:30.759049 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.758848 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\"" Sep 18 17:20:30.759049 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.758785 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\"" Sep 18 17:20:30.759049 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.758752 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 18 17:20:30.759337 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.759318 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 18 17:20:30.759693 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.759676 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 18 17:20:30.759784 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.759677 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\"" Sep 18 17:20:30.774048 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.774021 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6"] Sep 18 17:20:30.843819 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.843791 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0700380-27b2-4bfb-a7c7-a23a2843182f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.843988 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.843836 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.843988 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.843854 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8prls\" (UniqueName: \"kubernetes.io/projected/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-api-access-8prls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.843988 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.843877 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.843988 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.843901 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0700380-27b2-4bfb-a7c7-a23a2843182f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.945183 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.945145 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0700380-27b2-4bfb-a7c7-a23a2843182f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.945390 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.945195 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.945390 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.945215 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8prls\" (UniqueName: \"kubernetes.io/projected/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-api-access-8prls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.945390 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.945236 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.945390 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.945264 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0700380-27b2-4bfb-a7c7-a23a2843182f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.945680 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.945655 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0700380-27b2-4bfb-a7c7-a23a2843182f-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.946058 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.946030 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.946108 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.946066 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.947870 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.947848 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0700380-27b2-4bfb-a7c7-a23a2843182f-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:30.956788 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:30.956763 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8prls\" (UniqueName: \"kubernetes.io/projected/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-api-access-8prls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:31.062718 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:31.062620 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:31.344809 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:31.344727 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6"] Sep 18 17:20:31.348647 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:20:31.348616 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0700380_27b2_4bfb_a7c7_a23a2843182f.slice/crio-f837b333fb20d9926b58b2e59fd1cb893c0fddc57a8d7f31e3a70e4cc8da1e7f WatchSource:0}: Error finding container f837b333fb20d9926b58b2e59fd1cb893c0fddc57a8d7f31e3a70e4cc8da1e7f: Status 404 returned error can't find the container with id f837b333fb20d9926b58b2e59fd1cb893c0fddc57a8d7f31e3a70e4cc8da1e7f Sep 18 17:20:31.574173 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:31.574138 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" event={"ID":"f0700380-27b2-4bfb-a7c7-a23a2843182f","Type":"ContainerStarted","Data":"c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e"} Sep 18 17:20:31.574338 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:31.574207 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" event={"ID":"f0700380-27b2-4bfb-a7c7-a23a2843182f","Type":"ContainerStarted","Data":"f837b333fb20d9926b58b2e59fd1cb893c0fddc57a8d7f31e3a70e4cc8da1e7f"} Sep 18 17:20:52.673996 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:52.673954 2874 generic.go:356] "Generic (PLEG): container finished" podID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerID="c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e" exitCode=0 Sep 18 17:20:52.674389 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:52.674029 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" event={"ID":"f0700380-27b2-4bfb-a7c7-a23a2843182f","Type":"ContainerDied","Data":"c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e"} Sep 18 17:20:53.681135 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:53.681103 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" event={"ID":"f0700380-27b2-4bfb-a7c7-a23a2843182f","Type":"ContainerStarted","Data":"6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0"} Sep 18 17:20:53.681568 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:53.681161 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" event={"ID":"f0700380-27b2-4bfb-a7c7-a23a2843182f","Type":"ContainerStarted","Data":"78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d"} Sep 18 17:20:53.681568 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:53.681384 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:53.681568 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:53.681457 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:20:53.705031 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:53.704983 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" podStartSLOduration=3.443376055 podStartE2EDuration="23.704967906s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.261591851s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:53.702597254 +0000 UTC m=+1627.036961960" watchObservedRunningTime="2026-09-18 17:20:53.704967906 +0000 UTC m=+1627.039332613" Sep 18 17:20:59.691674 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:20:59.691642 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:22:25.776057 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:25.776027 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:22:36.785595 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:36.785546 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 18 17:22:37.742511 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:37.742481 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6"] Sep 18 17:22:37.742789 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:37.742751 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="kserve-container" containerID="cri-o://78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d" gracePeriod=30 Sep 18 17:22:37.742856 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:37.742780 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="kube-rbac-proxy" containerID="cri-o://6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0" gracePeriod=30 Sep 18 17:22:38.154128 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:38.154090 2874 generic.go:356] "Generic (PLEG): container finished" podID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerID="6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0" exitCode=2 Sep 18 17:22:38.154539 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:38.154134 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" event={"ID":"f0700380-27b2-4bfb-a7c7-a23a2843182f","Type":"ContainerDied","Data":"6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0"} Sep 18 17:22:39.093810 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.093784 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:22:39.161200 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.161161 2874 generic.go:356] "Generic (PLEG): container finished" podID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerID="78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d" exitCode=0 Sep 18 17:22:39.161640 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.161223 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" Sep 18 17:22:39.161640 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.161232 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" event={"ID":"f0700380-27b2-4bfb-a7c7-a23a2843182f","Type":"ContainerDied","Data":"78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d"} Sep 18 17:22:39.161640 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.161292 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6" event={"ID":"f0700380-27b2-4bfb-a7c7-a23a2843182f","Type":"ContainerDied","Data":"f837b333fb20d9926b58b2e59fd1cb893c0fddc57a8d7f31e3a70e4cc8da1e7f"} Sep 18 17:22:39.161640 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.161320 2874 scope.go:117] "RemoveContainer" containerID="6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0" Sep 18 17:22:39.170331 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.170314 2874 scope.go:117] "RemoveContainer" containerID="78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d" Sep 18 17:22:39.178379 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.178364 2874 scope.go:117] "RemoveContainer" containerID="c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e" Sep 18 17:22:39.186068 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.186054 2874 scope.go:117] "RemoveContainer" containerID="6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0" Sep 18 17:22:39.186315 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:22:39.186296 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0\": container with ID starting with 6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0 not found: ID does not exist" containerID="6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0" Sep 18 17:22:39.186374 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.186323 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0"} err="failed to get container status \"6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0\": rpc error: code = NotFound desc = could not find container \"6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0\": container with ID starting with 6203dde760bce20be57b83674a5d095f61542b139a08849ff932e28dc674d2b0 not found: ID does not exist" Sep 18 17:22:39.186374 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.186344 2874 scope.go:117] "RemoveContainer" containerID="78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d" Sep 18 17:22:39.186606 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:22:39.186589 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d\": container with ID starting with 78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d not found: ID does not exist" containerID="78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d" Sep 18 17:22:39.186650 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.186619 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d"} err="failed to get container status \"78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d\": rpc error: code = NotFound desc = could not find container \"78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d\": container with ID starting with 78bdd201f3c8d8644a67339b67b7ab41047597995bda4e5c7409fc0c123d433d not found: ID does not exist" Sep 18 17:22:39.186650 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.186633 2874 scope.go:117] "RemoveContainer" containerID="c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e" Sep 18 17:22:39.186847 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:22:39.186827 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e\": container with ID starting with c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e not found: ID does not exist" containerID="c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e" Sep 18 17:22:39.186885 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.186851 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e"} err="failed to get container status \"c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e\": rpc error: code = NotFound desc = could not find container \"c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e\": container with ID starting with c1bedaa94c211f77ca87944e96ee3ddabe4e04a8f17fa8bf425e88b1261bd37e not found: ID does not exist" Sep 18 17:22:39.235529 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.235454 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8prls\" (UniqueName: \"kubernetes.io/projected/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-api-access-8prls\") pod \"f0700380-27b2-4bfb-a7c7-a23a2843182f\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " Sep 18 17:22:39.235650 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.235542 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0700380-27b2-4bfb-a7c7-a23a2843182f-kserve-provision-location\") pod \"f0700380-27b2-4bfb-a7c7-a23a2843182f\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " Sep 18 17:22:39.235650 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.235597 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0700380-27b2-4bfb-a7c7-a23a2843182f-proxy-tls\") pod \"f0700380-27b2-4bfb-a7c7-a23a2843182f\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " Sep 18 17:22:39.235764 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.235655 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-cabundle-cert\") pod \"f0700380-27b2-4bfb-a7c7-a23a2843182f\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " Sep 18 17:22:39.235764 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.235683 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-rbac-proxy-sar-config\") pod \"f0700380-27b2-4bfb-a7c7-a23a2843182f\" (UID: \"f0700380-27b2-4bfb-a7c7-a23a2843182f\") " Sep 18 17:22:39.236049 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.236023 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0700380-27b2-4bfb-a7c7-a23a2843182f" (UID: "f0700380-27b2-4bfb-a7c7-a23a2843182f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:22:39.236144 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.236120 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0700380-27b2-4bfb-a7c7-a23a2843182f" (UID: "f0700380-27b2-4bfb-a7c7-a23a2843182f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:22:39.237754 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.237730 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-api-access-8prls" (OuterVolumeSpecName: "kube-api-access-8prls") pod "f0700380-27b2-4bfb-a7c7-a23a2843182f" (UID: "f0700380-27b2-4bfb-a7c7-a23a2843182f"). InnerVolumeSpecName "kube-api-access-8prls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:22:39.237830 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.237794 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0700380-27b2-4bfb-a7c7-a23a2843182f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0700380-27b2-4bfb-a7c7-a23a2843182f" (UID: "f0700380-27b2-4bfb-a7c7-a23a2843182f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:22:39.241918 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.241893 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0700380-27b2-4bfb-a7c7-a23a2843182f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0700380-27b2-4bfb-a7c7-a23a2843182f" (UID: "f0700380-27b2-4bfb-a7c7-a23a2843182f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:22:39.336895 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.336859 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0700380-27b2-4bfb-a7c7-a23a2843182f-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:22:39.336895 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.336888 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0700380-27b2-4bfb-a7c7-a23a2843182f-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:22:39.336895 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.336898 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:22:39.337123 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.336906 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:22:39.337123 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.336916 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8prls\" (UniqueName: \"kubernetes.io/projected/f0700380-27b2-4bfb-a7c7-a23a2843182f-kube-api-access-8prls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:22:39.493045 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.492968 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6"] Sep 18 17:22:39.499075 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:39.499047 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dc2w6"] Sep 18 17:22:40.749064 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:40.749031 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" path="/var/lib/kubelet/pods/f0700380-27b2-4bfb-a7c7-a23a2843182f/volumes" Sep 18 17:22:43.006625 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.006587 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc"] Sep 18 17:22:43.007167 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.007004 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="kube-rbac-proxy" Sep 18 17:22:43.007167 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.007019 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="kube-rbac-proxy" Sep 18 17:22:43.007167 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.007067 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="kserve-container" Sep 18 17:22:43.007167 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.007076 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="kserve-container" Sep 18 17:22:43.007167 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.007094 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="storage-initializer" Sep 18 17:22:43.007167 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.007101 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="storage-initializer" Sep 18 17:22:43.007473 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.007194 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="kserve-container" Sep 18 17:22:43.007473 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.007212 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0700380-27b2-4bfb-a7c7-a23a2843182f" containerName="kube-rbac-proxy" Sep 18 17:22:43.010695 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.010675 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.019341 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.019301 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 18 17:22:43.019443 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.019325 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\"" Sep 18 17:22:43.020220 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.020199 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\"" Sep 18 17:22:43.021554 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.021518 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 18 17:22:43.021643 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.021542 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 18 17:22:43.021714 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.021680 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\"" Sep 18 17:22:43.021773 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.021715 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 18 17:22:43.049824 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.049794 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc"] Sep 18 17:22:43.169777 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.169742 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.169930 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.169801 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbz66\" (UniqueName: \"kubernetes.io/projected/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-api-access-bbz66\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.169930 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.169836 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.169930 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.169856 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/608d706a-2204-45d1-95e0-47d9d5baaef0-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.170046 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.169952 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/608d706a-2204-45d1-95e0-47d9d5baaef0-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.270763 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.270683 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.270763 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.270732 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbz66\" (UniqueName: \"kubernetes.io/projected/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-api-access-bbz66\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.270763 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.270755 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.271045 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.270781 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/608d706a-2204-45d1-95e0-47d9d5baaef0-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.271045 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.270828 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/608d706a-2204-45d1-95e0-47d9d5baaef0-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.271223 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.271203 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/608d706a-2204-45d1-95e0-47d9d5baaef0-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.271486 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.271462 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.271601 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.271463 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.273391 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.273375 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/608d706a-2204-45d1-95e0-47d9d5baaef0-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.284672 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.284653 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbz66\" (UniqueName: \"kubernetes.io/projected/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-api-access-bbz66\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.320577 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.320546 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:22:43.611870 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:43.611816 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc"] Sep 18 17:22:43.614510 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:22:43.614477 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608d706a_2204_45d1_95e0_47d9d5baaef0.slice/crio-3eaaa738d9692287097239a9ce0c745791c5f08cd2dded0224f0e0341497e072 WatchSource:0}: Error finding container 3eaaa738d9692287097239a9ce0c745791c5f08cd2dded0224f0e0341497e072: Status 404 returned error can't find the container with id 3eaaa738d9692287097239a9ce0c745791c5f08cd2dded0224f0e0341497e072 Sep 18 17:22:44.185858 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:44.185810 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" event={"ID":"608d706a-2204-45d1-95e0-47d9d5baaef0","Type":"ContainerStarted","Data":"90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923"} Sep 18 17:22:44.186321 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:22:44.185881 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" event={"ID":"608d706a-2204-45d1-95e0-47d9d5baaef0","Type":"ContainerStarted","Data":"3eaaa738d9692287097239a9ce0c745791c5f08cd2dded0224f0e0341497e072"} Sep 18 17:23:13.320745 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:23:13.320706 2874 generic.go:356] "Generic (PLEG): container finished" podID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerID="90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923" exitCode=0 Sep 18 17:23:13.321176 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:23:13.320782 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" event={"ID":"608d706a-2204-45d1-95e0-47d9d5baaef0","Type":"ContainerDied","Data":"90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923"} Sep 18 17:23:14.331984 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:23:14.331947 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" event={"ID":"608d706a-2204-45d1-95e0-47d9d5baaef0","Type":"ContainerStarted","Data":"66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a"} Sep 18 17:23:14.332483 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:23:14.332048 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" event={"ID":"608d706a-2204-45d1-95e0-47d9d5baaef0","Type":"ContainerStarted","Data":"b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd"} Sep 18 17:23:14.332483 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:23:14.332282 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:23:14.364534 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:23:14.364488 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" podStartSLOduration=2.934992529 podStartE2EDuration="32.364472493s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.429479964s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:23:14.363964841 +0000 UTC m=+1767.698329562" watchObservedRunningTime="2026-09-18 17:23:14.364472493 +0000 UTC m=+1767.698837212" Sep 18 17:23:15.336566 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:23:15.336533 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:23:21.347409 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:23:21.347379 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:23:46.910222 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:23:46.910188 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:23:46.911643 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:23:46.910801 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:24:51.778437 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:24:51.778321 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:24:59.131093 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:24:59.131055 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc"] Sep 18 17:24:59.132013 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:24:59.131978 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerName="kserve-container" containerID="cri-o://b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd" gracePeriod=30 Sep 18 17:24:59.135428 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:24:59.132386 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerName="kube-rbac-proxy" containerID="cri-o://66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a" gracePeriod=30 Sep 18 17:24:59.827123 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:24:59.827089 2874 generic.go:356] "Generic (PLEG): container finished" podID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerID="66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a" exitCode=2 Sep 18 17:24:59.827298 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:24:59.827161 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" event={"ID":"608d706a-2204-45d1-95e0-47d9d5baaef0","Type":"ContainerDied","Data":"66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a"} Sep 18 17:25:00.473034 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.473009 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:25:00.516277 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.516249 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/608d706a-2204-45d1-95e0-47d9d5baaef0-kserve-provision-location\") pod \"608d706a-2204-45d1-95e0-47d9d5baaef0\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " Sep 18 17:25:00.516395 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.516287 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-rbac-proxy-sar-config\") pod \"608d706a-2204-45d1-95e0-47d9d5baaef0\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " Sep 18 17:25:00.516395 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.516319 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-cabundle-cert\") pod \"608d706a-2204-45d1-95e0-47d9d5baaef0\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " Sep 18 17:25:00.516535 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.516512 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbz66\" (UniqueName: \"kubernetes.io/projected/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-api-access-bbz66\") pod \"608d706a-2204-45d1-95e0-47d9d5baaef0\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " Sep 18 17:25:00.516616 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.516553 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/608d706a-2204-45d1-95e0-47d9d5baaef0-proxy-tls\") pod \"608d706a-2204-45d1-95e0-47d9d5baaef0\" (UID: \"608d706a-2204-45d1-95e0-47d9d5baaef0\") " Sep 18 17:25:00.516697 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.516669 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "608d706a-2204-45d1-95e0-47d9d5baaef0" (UID: "608d706a-2204-45d1-95e0-47d9d5baaef0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:25:00.516783 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.516676 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "608d706a-2204-45d1-95e0-47d9d5baaef0" (UID: "608d706a-2204-45d1-95e0-47d9d5baaef0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:25:00.516881 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.516860 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:25:00.516958 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.516887 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/608d706a-2204-45d1-95e0-47d9d5baaef0-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:25:00.518925 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.518900 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-api-access-bbz66" (OuterVolumeSpecName: "kube-api-access-bbz66") pod "608d706a-2204-45d1-95e0-47d9d5baaef0" (UID: "608d706a-2204-45d1-95e0-47d9d5baaef0"). InnerVolumeSpecName "kube-api-access-bbz66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:25:00.519125 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.519110 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/608d706a-2204-45d1-95e0-47d9d5baaef0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "608d706a-2204-45d1-95e0-47d9d5baaef0" (UID: "608d706a-2204-45d1-95e0-47d9d5baaef0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:25:00.520536 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.520514 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608d706a-2204-45d1-95e0-47d9d5baaef0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "608d706a-2204-45d1-95e0-47d9d5baaef0" (UID: "608d706a-2204-45d1-95e0-47d9d5baaef0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:25:00.617472 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.617363 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/608d706a-2204-45d1-95e0-47d9d5baaef0-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:25:00.617472 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.617394 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbz66\" (UniqueName: \"kubernetes.io/projected/608d706a-2204-45d1-95e0-47d9d5baaef0-kube-api-access-bbz66\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:25:00.617472 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.617435 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/608d706a-2204-45d1-95e0-47d9d5baaef0-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:25:00.833464 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.833434 2874 generic.go:356] "Generic (PLEG): container finished" podID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerID="b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd" exitCode=0 Sep 18 17:25:00.833617 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.833486 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" event={"ID":"608d706a-2204-45d1-95e0-47d9d5baaef0","Type":"ContainerDied","Data":"b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd"} Sep 18 17:25:00.833617 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.833493 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" Sep 18 17:25:00.833617 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.833535 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc" event={"ID":"608d706a-2204-45d1-95e0-47d9d5baaef0","Type":"ContainerDied","Data":"3eaaa738d9692287097239a9ce0c745791c5f08cd2dded0224f0e0341497e072"} Sep 18 17:25:00.833617 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.833565 2874 scope.go:117] "RemoveContainer" containerID="66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a" Sep 18 17:25:00.841958 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.841939 2874 scope.go:117] "RemoveContainer" containerID="b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd" Sep 18 17:25:00.849599 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.849583 2874 scope.go:117] "RemoveContainer" containerID="90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923" Sep 18 17:25:00.857021 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.857006 2874 scope.go:117] "RemoveContainer" containerID="66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a" Sep 18 17:25:00.857292 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:25:00.857266 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a\": container with ID starting with 66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a not found: ID does not exist" containerID="66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a" Sep 18 17:25:00.857348 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.857292 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a"} err="failed to get container status \"66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a\": rpc error: code = NotFound desc = could not find container \"66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a\": container with ID starting with 66eeda5b828964b1e15218eb8fd46583649f43c403602a036535472c71e9379a not found: ID does not exist" Sep 18 17:25:00.857348 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.857307 2874 scope.go:117] "RemoveContainer" containerID="b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd" Sep 18 17:25:00.857574 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:25:00.857558 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd\": container with ID starting with b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd not found: ID does not exist" containerID="b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd" Sep 18 17:25:00.857625 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.857579 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd"} err="failed to get container status \"b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd\": rpc error: code = NotFound desc = could not find container \"b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd\": container with ID starting with b63142d961882706b21fc4ff25dcf8e9719597a2460a9daf4b9fc0f6f64a3bbd not found: ID does not exist" Sep 18 17:25:00.857625 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.857592 2874 scope.go:117] "RemoveContainer" containerID="90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923" Sep 18 17:25:00.857815 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:25:00.857797 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923\": container with ID starting with 90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923 not found: ID does not exist" containerID="90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923" Sep 18 17:25:00.857873 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.857824 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923"} err="failed to get container status \"90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923\": rpc error: code = NotFound desc = could not find container \"90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923\": container with ID starting with 90f44486124a8d14b7b3770a272910537a4afd2846605f1415657ba877fcc923 not found: ID does not exist" Sep 18 17:25:00.872206 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.872152 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc"] Sep 18 17:25:00.877531 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:00.877510 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-8xvbc"] Sep 18 17:25:02.748932 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:02.748899 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" path="/var/lib/kubelet/pods/608d706a-2204-45d1-95e0-47d9d5baaef0/volumes" Sep 18 17:25:04.562847 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.562809 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm"] Sep 18 17:25:04.563322 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.563156 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerName="kserve-container" Sep 18 17:25:04.563322 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.563168 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerName="kserve-container" Sep 18 17:25:04.563322 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.563176 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerName="storage-initializer" Sep 18 17:25:04.563322 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.563182 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerName="storage-initializer" Sep 18 17:25:04.563322 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.563190 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerName="kube-rbac-proxy" Sep 18 17:25:04.563322 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.563196 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerName="kube-rbac-proxy" Sep 18 17:25:04.563322 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.563246 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerName="kserve-container" Sep 18 17:25:04.563322 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.563255 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="608d706a-2204-45d1-95e0-47d9d5baaef0" containerName="kube-rbac-proxy" Sep 18 17:25:04.567807 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.567786 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.578609 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.578589 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 18 17:25:04.578804 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.578784 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\"" Sep 18 17:25:04.578921 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.578811 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 18 17:25:04.578921 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.578906 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 18 17:25:04.579026 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.578939 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 18 17:25:04.579661 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.579648 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\"" Sep 18 17:25:04.589026 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.589009 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\"" Sep 18 17:25:04.590473 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.590453 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm"] Sep 18 17:25:04.645624 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.645600 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.645759 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.645641 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.645759 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.645669 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2vfb\" (UniqueName: \"kubernetes.io/projected/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-api-access-q2vfb\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.645843 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.645774 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.645843 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.645804 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.746235 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.746203 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.746235 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.746232 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2vfb\" (UniqueName: \"kubernetes.io/projected/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-api-access-q2vfb\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.746541 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.746281 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.746541 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.746303 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.746541 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.746443 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.746749 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.746721 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.747100 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.747078 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.747190 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.747118 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.748828 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.748810 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.760355 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.760330 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2vfb\" (UniqueName: \"kubernetes.io/projected/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-api-access-q2vfb\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:04.878214 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:04.878189 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:05.171206 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:05.171184 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm"] Sep 18 17:25:05.173106 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:25:05.173083 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9ff02c_06ce_486a_89c7_a7ce797fdb75.slice/crio-c5f00f1b79e62bfe837377cb5775ceab424e3ff588e4e4d18eb60ba7b503c201 WatchSource:0}: Error finding container c5f00f1b79e62bfe837377cb5775ceab424e3ff588e4e4d18eb60ba7b503c201: Status 404 returned error can't find the container with id c5f00f1b79e62bfe837377cb5775ceab424e3ff588e4e4d18eb60ba7b503c201 Sep 18 17:25:05.858387 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:05.858337 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" event={"ID":"8d9ff02c-06ce-486a-89c7-a7ce797fdb75","Type":"ContainerStarted","Data":"b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb"} Sep 18 17:25:05.858872 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:05.858440 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" event={"ID":"8d9ff02c-06ce-486a-89c7-a7ce797fdb75","Type":"ContainerStarted","Data":"c5f00f1b79e62bfe837377cb5775ceab424e3ff588e4e4d18eb60ba7b503c201"} Sep 18 17:25:14.901384 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:14.901348 2874 generic.go:356] "Generic (PLEG): container finished" podID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerID="b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb" exitCode=0 Sep 18 17:25:14.901771 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:14.901435 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" event={"ID":"8d9ff02c-06ce-486a-89c7-a7ce797fdb75","Type":"ContainerDied","Data":"b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb"} Sep 18 17:25:15.909219 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:15.909175 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" event={"ID":"8d9ff02c-06ce-486a-89c7-a7ce797fdb75","Type":"ContainerStarted","Data":"0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0"} Sep 18 17:25:15.909670 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:15.909255 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" event={"ID":"8d9ff02c-06ce-486a-89c7-a7ce797fdb75","Type":"ContainerStarted","Data":"2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0"} Sep 18 17:25:15.909670 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:15.909508 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:15.931805 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:15.931752 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" podStartSLOduration=2.814670611 podStartE2EDuration="11.931738169s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.117067558s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:25:15.929691659 +0000 UTC m=+1889.264056389" watchObservedRunningTime="2026-09-18 17:25:15.931738169 +0000 UTC m=+1889.266102876" Sep 18 17:25:16.913991 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:16.913957 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:16.915756 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:16.915725 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 18 17:25:17.918620 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:17.918578 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 18 17:25:22.923540 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:22.923500 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:22.924082 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:22.924056 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 18 17:25:32.925601 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:32.925572 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:46.789532 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:46.789450 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm"] Sep 18 17:25:46.789951 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:46.789742 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kserve-container" containerID="cri-o://2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0" gracePeriod=30 Sep 18 17:25:46.789951 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:46.789826 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kube-rbac-proxy" containerID="cri-o://0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0" gracePeriod=30 Sep 18 17:25:47.058301 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:47.058204 2874 generic.go:356] "Generic (PLEG): container finished" podID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerID="0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0" exitCode=2 Sep 18 17:25:47.058301 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:47.058278 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" event={"ID":"8d9ff02c-06ce-486a-89c7-a7ce797fdb75","Type":"ContainerDied","Data":"0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0"} Sep 18 17:25:47.919777 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:47.919717 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Sep 18 17:25:48.537876 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.537854 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:48.630213 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.630178 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2vfb\" (UniqueName: \"kubernetes.io/projected/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-api-access-q2vfb\") pod \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " Sep 18 17:25:48.630452 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.630234 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-rbac-proxy-sar-config\") pod \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " Sep 18 17:25:48.630452 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.630287 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-cabundle-cert\") pod \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " Sep 18 17:25:48.630452 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.630329 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-proxy-tls\") pod \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " Sep 18 17:25:48.630452 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.630357 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kserve-provision-location\") pod \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\" (UID: \"8d9ff02c-06ce-486a-89c7-a7ce797fdb75\") " Sep 18 17:25:48.630790 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.630759 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8d9ff02c-06ce-486a-89c7-a7ce797fdb75" (UID: "8d9ff02c-06ce-486a-89c7-a7ce797fdb75"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:25:48.630875 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.630798 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d9ff02c-06ce-486a-89c7-a7ce797fdb75" (UID: "8d9ff02c-06ce-486a-89c7-a7ce797fdb75"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:25:48.631504 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.631463 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d9ff02c-06ce-486a-89c7-a7ce797fdb75" (UID: "8d9ff02c-06ce-486a-89c7-a7ce797fdb75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:25:48.632760 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.632738 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-api-access-q2vfb" (OuterVolumeSpecName: "kube-api-access-q2vfb") pod "8d9ff02c-06ce-486a-89c7-a7ce797fdb75" (UID: "8d9ff02c-06ce-486a-89c7-a7ce797fdb75"). InnerVolumeSpecName "kube-api-access-q2vfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:25:48.632847 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.632831 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d9ff02c-06ce-486a-89c7-a7ce797fdb75" (UID: "8d9ff02c-06ce-486a-89c7-a7ce797fdb75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:25:48.731245 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.731214 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2vfb\" (UniqueName: \"kubernetes.io/projected/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-api-access-q2vfb\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:25:48.731245 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.731240 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:25:48.731245 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.731251 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:25:48.731482 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.731260 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:25:48.731482 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:48.731268 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9ff02c-06ce-486a-89c7-a7ce797fdb75-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:25:49.069738 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.069646 2874 generic.go:356] "Generic (PLEG): container finished" podID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerID="2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0" exitCode=0 Sep 18 17:25:49.069738 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.069706 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" Sep 18 17:25:49.069738 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.069701 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" event={"ID":"8d9ff02c-06ce-486a-89c7-a7ce797fdb75","Type":"ContainerDied","Data":"2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0"} Sep 18 17:25:49.070248 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.069769 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm" event={"ID":"8d9ff02c-06ce-486a-89c7-a7ce797fdb75","Type":"ContainerDied","Data":"c5f00f1b79e62bfe837377cb5775ceab424e3ff588e4e4d18eb60ba7b503c201"} Sep 18 17:25:49.070248 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.069794 2874 scope.go:117] "RemoveContainer" containerID="0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0" Sep 18 17:25:49.078805 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.078787 2874 scope.go:117] "RemoveContainer" containerID="2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0" Sep 18 17:25:49.086672 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.086649 2874 scope.go:117] "RemoveContainer" containerID="b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb" Sep 18 17:25:49.090350 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.090329 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm"] Sep 18 17:25:49.094737 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.094721 2874 scope.go:117] "RemoveContainer" containerID="0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0" Sep 18 17:25:49.094932 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.094912 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m8qmm"] Sep 18 17:25:49.095030 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:25:49.095011 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0\": container with ID starting with 0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0 not found: ID does not exist" containerID="0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0" Sep 18 17:25:49.095075 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.095038 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0"} err="failed to get container status \"0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0\": rpc error: code = NotFound desc = could not find container \"0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0\": container with ID starting with 0d8ca57301cfbdfb49490b7741fd88f251c8cf6560b0d12a343f0493b22704a0 not found: ID does not exist" Sep 18 17:25:49.095075 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.095055 2874 scope.go:117] "RemoveContainer" containerID="2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0" Sep 18 17:25:49.095298 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:25:49.095281 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0\": container with ID starting with 2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0 not found: ID does not exist" containerID="2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0" Sep 18 17:25:49.095363 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.095307 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0"} err="failed to get container status \"2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0\": rpc error: code = NotFound desc = could not find container \"2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0\": container with ID starting with 2d32c302fd32dfc5533321091089ee4b867a494421bd33758a9fcb1230475fd0 not found: ID does not exist" Sep 18 17:25:49.095363 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.095330 2874 scope.go:117] "RemoveContainer" containerID="b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb" Sep 18 17:25:49.095602 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:25:49.095584 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb\": container with ID starting with b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb not found: ID does not exist" containerID="b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb" Sep 18 17:25:49.095650 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:49.095609 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb"} err="failed to get container status \"b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb\": rpc error: code = NotFound desc = could not find container \"b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb\": container with ID starting with b6cfbbd357b0f13407bb34a388fc8673f176f80113e4e5f3b80b28bfd50547fb not found: ID does not exist" Sep 18 17:25:50.748762 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:50.748729 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" path="/var/lib/kubelet/pods/8d9ff02c-06ce-486a-89c7-a7ce797fdb75/volumes" Sep 18 17:25:52.001772 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.001735 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs"] Sep 18 17:25:52.002242 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.002207 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="storage-initializer" Sep 18 17:25:52.002242 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.002227 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="storage-initializer" Sep 18 17:25:52.002372 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.002246 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kserve-container" Sep 18 17:25:52.002372 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.002257 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kserve-container" Sep 18 17:25:52.002372 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.002276 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kube-rbac-proxy" Sep 18 17:25:52.002372 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.002285 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kube-rbac-proxy" Sep 18 17:25:52.002606 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.002382 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kube-rbac-proxy" Sep 18 17:25:52.002606 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.002418 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9ff02c-06ce-486a-89c7-a7ce797fdb75" containerName="kserve-container" Sep 18 17:25:52.007669 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.007646 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.011194 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.011171 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\"" Sep 18 17:25:52.011309 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.011208 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\"" Sep 18 17:25:52.011492 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.011475 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 18 17:25:52.011600 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.011477 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 18 17:25:52.011725 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.011707 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\"" Sep 18 17:25:52.011766 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.011723 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 18 17:25:52.012225 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.012212 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 18 17:25:52.025077 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.025044 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs"] Sep 18 17:25:52.159198 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.159167 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e23e6324-4115-443a-a166-0e05e21962e3-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.159383 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.159208 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e23e6324-4115-443a-a166-0e05e21962e3-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.159383 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.159228 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.159383 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.159250 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.159383 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.159306 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvrm\" (UniqueName: \"kubernetes.io/projected/e23e6324-4115-443a-a166-0e05e21962e3-kube-api-access-pmvrm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.260663 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.260577 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e23e6324-4115-443a-a166-0e05e21962e3-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.260663 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.260621 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e23e6324-4115-443a-a166-0e05e21962e3-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.260663 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.260637 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.260663 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.260659 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.260990 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.260682 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvrm\" (UniqueName: \"kubernetes.io/projected/e23e6324-4115-443a-a166-0e05e21962e3-kube-api-access-pmvrm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.260990 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:25:52.260777 2874 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found Sep 18 17:25:52.260990 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:25:52.260853 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e23e6324-4115-443a-a166-0e05e21962e3-proxy-tls podName:e23e6324-4115-443a-a166-0e05e21962e3 nodeName:}" failed. No retries permitted until 2026-09-18 17:25:52.760832659 +0000 UTC m=+1926.095197362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e23e6324-4115-443a-a166-0e05e21962e3-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" (UID: "e23e6324-4115-443a-a166-0e05e21962e3") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found Sep 18 17:25:52.261172 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.261032 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e23e6324-4115-443a-a166-0e05e21962e3-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.261426 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.261347 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.261511 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.261439 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.271812 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.271793 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvrm\" (UniqueName: \"kubernetes.io/projected/e23e6324-4115-443a-a166-0e05e21962e3-kube-api-access-pmvrm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.765698 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.765659 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e23e6324-4115-443a-a166-0e05e21962e3-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.768213 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.768191 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e23e6324-4115-443a-a166-0e05e21962e3-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:52.924218 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:52.924185 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:25:53.204415 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:53.204370 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs"] Sep 18 17:25:53.207188 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:25:53.207163 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23e6324_4115_443a_a166_0e05e21962e3.slice/crio-1940e3f368eb706ca10d24b8b01b3305d72d6e9ea85fe4135307abb125b6f4b2 WatchSource:0}: Error finding container 1940e3f368eb706ca10d24b8b01b3305d72d6e9ea85fe4135307abb125b6f4b2: Status 404 returned error can't find the container with id 1940e3f368eb706ca10d24b8b01b3305d72d6e9ea85fe4135307abb125b6f4b2 Sep 18 17:25:54.095711 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:54.095680 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" event={"ID":"e23e6324-4115-443a-a166-0e05e21962e3","Type":"ContainerStarted","Data":"f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0"} Sep 18 17:25:54.096019 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:25:54.095738 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" event={"ID":"e23e6324-4115-443a-a166-0e05e21962e3","Type":"ContainerStarted","Data":"1940e3f368eb706ca10d24b8b01b3305d72d6e9ea85fe4135307abb125b6f4b2"} Sep 18 17:26:02.134006 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:02.133973 2874 generic.go:356] "Generic (PLEG): container finished" podID="e23e6324-4115-443a-a166-0e05e21962e3" containerID="f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0" exitCode=0 Sep 18 17:26:02.134379 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:02.134045 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" event={"ID":"e23e6324-4115-443a-a166-0e05e21962e3","Type":"ContainerDied","Data":"f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0"} Sep 18 17:26:03.140592 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:03.140558 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" event={"ID":"e23e6324-4115-443a-a166-0e05e21962e3","Type":"ContainerStarted","Data":"6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275"} Sep 18 17:26:03.141112 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:03.140617 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" event={"ID":"e23e6324-4115-443a-a166-0e05e21962e3","Type":"ContainerStarted","Data":"cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2"} Sep 18 17:26:03.141112 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:03.140800 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:26:03.164050 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:03.163996 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" podStartSLOduration=3.667110771 podStartE2EDuration="12.163980713s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.496869942s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:03.162807645 +0000 UTC m=+1936.497172355" watchObservedRunningTime="2026-09-18 17:26:03.163980713 +0000 UTC m=+1936.498345420" Sep 18 17:26:04.146484 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:04.146434 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:26:04.148105 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:04.148070 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 18 17:26:05.150950 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:05.150912 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 18 17:26:10.155638 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:10.155602 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:26:10.156333 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:10.156306 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Sep 18 17:26:20.157211 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:20.157182 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:26:27.283751 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:27.283721 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs"] Sep 18 17:26:27.284315 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:27.284027 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kserve-container" containerID="cri-o://cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2" gracePeriod=30 Sep 18 17:26:27.284315 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:27.284051 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kube-rbac-proxy" containerID="cri-o://6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275" gracePeriod=30 Sep 18 17:26:28.262352 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.262319 2874 generic.go:356] "Generic (PLEG): container finished" podID="e23e6324-4115-443a-a166-0e05e21962e3" containerID="6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275" exitCode=2 Sep 18 17:26:28.262562 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.262366 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" event={"ID":"e23e6324-4115-443a-a166-0e05e21962e3","Type":"ContainerDied","Data":"6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275"} Sep 18 17:26:28.730683 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.730660 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:26:28.861134 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.861056 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e23e6324-4115-443a-a166-0e05e21962e3-proxy-tls\") pod \"e23e6324-4115-443a-a166-0e05e21962e3\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " Sep 18 17:26:28.861134 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.861112 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e23e6324-4115-443a-a166-0e05e21962e3-kserve-provision-location\") pod \"e23e6324-4115-443a-a166-0e05e21962e3\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " Sep 18 17:26:28.861343 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.861146 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmvrm\" (UniqueName: \"kubernetes.io/projected/e23e6324-4115-443a-a166-0e05e21962e3-kube-api-access-pmvrm\") pod \"e23e6324-4115-443a-a166-0e05e21962e3\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " Sep 18 17:26:28.861343 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.861193 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-cabundle-cert\") pod \"e23e6324-4115-443a-a166-0e05e21962e3\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " Sep 18 17:26:28.861343 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.861215 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-kube-rbac-proxy-sar-config\") pod \"e23e6324-4115-443a-a166-0e05e21962e3\" (UID: \"e23e6324-4115-443a-a166-0e05e21962e3\") " Sep 18 17:26:28.861638 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.861614 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e23e6324-4115-443a-a166-0e05e21962e3" (UID: "e23e6324-4115-443a-a166-0e05e21962e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:26:28.861705 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.861662 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e23e6324-4115-443a-a166-0e05e21962e3" (UID: "e23e6324-4115-443a-a166-0e05e21962e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:26:28.862229 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.862210 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23e6324-4115-443a-a166-0e05e21962e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e23e6324-4115-443a-a166-0e05e21962e3" (UID: "e23e6324-4115-443a-a166-0e05e21962e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:26:28.863467 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.863443 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23e6324-4115-443a-a166-0e05e21962e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e23e6324-4115-443a-a166-0e05e21962e3" (UID: "e23e6324-4115-443a-a166-0e05e21962e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:26:28.863577 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.863516 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23e6324-4115-443a-a166-0e05e21962e3-kube-api-access-pmvrm" (OuterVolumeSpecName: "kube-api-access-pmvrm") pod "e23e6324-4115-443a-a166-0e05e21962e3" (UID: "e23e6324-4115-443a-a166-0e05e21962e3"). InnerVolumeSpecName "kube-api-access-pmvrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:26:28.962656 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.962619 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:26:28.962656 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.962655 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e23e6324-4115-443a-a166-0e05e21962e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:26:28.962656 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.962668 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e23e6324-4115-443a-a166-0e05e21962e3-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:26:28.962919 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.962682 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e23e6324-4115-443a-a166-0e05e21962e3-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:26:28.962919 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:28.962694 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmvrm\" (UniqueName: \"kubernetes.io/projected/e23e6324-4115-443a-a166-0e05e21962e3-kube-api-access-pmvrm\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:26:29.276215 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.276177 2874 generic.go:356] "Generic (PLEG): container finished" podID="e23e6324-4115-443a-a166-0e05e21962e3" containerID="cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2" exitCode=0 Sep 18 17:26:29.276381 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.276226 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" event={"ID":"e23e6324-4115-443a-a166-0e05e21962e3","Type":"ContainerDied","Data":"cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2"} Sep 18 17:26:29.276381 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.276261 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" Sep 18 17:26:29.276381 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.276319 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs" event={"ID":"e23e6324-4115-443a-a166-0e05e21962e3","Type":"ContainerDied","Data":"1940e3f368eb706ca10d24b8b01b3305d72d6e9ea85fe4135307abb125b6f4b2"} Sep 18 17:26:29.276381 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.276360 2874 scope.go:117] "RemoveContainer" containerID="6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275" Sep 18 17:26:29.285453 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.285436 2874 scope.go:117] "RemoveContainer" containerID="cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2" Sep 18 17:26:29.293359 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.293341 2874 scope.go:117] "RemoveContainer" containerID="f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0" Sep 18 17:26:29.299899 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.299874 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs"] Sep 18 17:26:29.301718 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.301698 2874 scope.go:117] "RemoveContainer" containerID="6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275" Sep 18 17:26:29.302243 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:26:29.302187 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275\": container with ID starting with 6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275 not found: ID does not exist" containerID="6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275" Sep 18 17:26:29.302364 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.302234 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275"} err="failed to get container status \"6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275\": rpc error: code = NotFound desc = could not find container \"6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275\": container with ID starting with 6cd6f222776c1a57e364a8f9eb00e670641a568a22bd8aa630ab9b5f5fc2a275 not found: ID does not exist" Sep 18 17:26:29.302364 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.302260 2874 scope.go:117] "RemoveContainer" containerID="cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2" Sep 18 17:26:29.302922 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:26:29.302899 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2\": container with ID starting with cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2 not found: ID does not exist" containerID="cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2" Sep 18 17:26:29.303018 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.302931 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2"} err="failed to get container status \"cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2\": rpc error: code = NotFound desc = could not find container \"cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2\": container with ID starting with cc8dc2db3e0b5d031e27c79d7ed9844cdf7ff0d5f519623c6849aedacf9e1ed2 not found: ID does not exist" Sep 18 17:26:29.303018 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.302953 2874 scope.go:117] "RemoveContainer" containerID="f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0" Sep 18 17:26:29.303274 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:26:29.303253 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0\": container with ID starting with f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0 not found: ID does not exist" containerID="f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0" Sep 18 17:26:29.303352 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.303285 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0"} err="failed to get container status \"f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0\": rpc error: code = NotFound desc = could not find container \"f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0\": container with ID starting with f564d86d8836a84c138b28693a2403c94a37e3206f239fe13e7fe066fd8d30d0 not found: ID does not exist" Sep 18 17:26:29.304875 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:29.304855 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-zc2gs"] Sep 18 17:26:30.748815 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:30.748774 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e23e6324-4115-443a-a166-0e05e21962e3" path="/var/lib/kubelet/pods/e23e6324-4115-443a-a166-0e05e21962e3/volumes" Sep 18 17:26:32.618436 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.618386 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl"] Sep 18 17:26:32.618827 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.618773 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kube-rbac-proxy" Sep 18 17:26:32.618827 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.618785 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kube-rbac-proxy" Sep 18 17:26:32.618827 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.618796 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kserve-container" Sep 18 17:26:32.618827 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.618804 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kserve-container" Sep 18 17:26:32.618827 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.618816 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="storage-initializer" Sep 18 17:26:32.618827 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.618824 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="storage-initializer" Sep 18 17:26:32.619055 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.618873 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kserve-container" Sep 18 17:26:32.619055 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.618887 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="e23e6324-4115-443a-a166-0e05e21962e3" containerName="kube-rbac-proxy" Sep 18 17:26:32.623479 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.623457 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.626031 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.626010 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\"" Sep 18 17:26:32.626939 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.626923 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 18 17:26:32.627060 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.626960 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\"" Sep 18 17:26:32.627420 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.627377 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 18 17:26:32.627653 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.627637 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 18 17:26:32.627865 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.627849 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" Sep 18 17:26:32.627968 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.627953 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 18 17:26:32.634193 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.634170 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl"] Sep 18 17:26:32.692670 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.692628 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdztq\" (UniqueName: \"kubernetes.io/projected/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-api-access-qdztq\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.692850 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.692709 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.692850 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.692748 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.692850 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.692834 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3f9c0-c03a-4dbf-a1d6-584000bed761-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.692956 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.692873 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.793674 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.793640 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3f9c0-c03a-4dbf-a1d6-584000bed761-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.793674 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.793678 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.793922 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.793732 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdztq\" (UniqueName: \"kubernetes.io/projected/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-api-access-qdztq\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.793922 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.793751 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.793922 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.793767 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.794219 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.794195 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.794554 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.794536 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.794610 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.794551 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.796320 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.796297 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3f9c0-c03a-4dbf-a1d6-584000bed761-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.803132 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.803111 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdztq\" (UniqueName: \"kubernetes.io/projected/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-api-access-qdztq\") pod \"isvc-canary-create-predictor-b75cfc78d-8qfbl\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:32.934641 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:32.934615 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:33.212021 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:33.211947 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl"] Sep 18 17:26:33.214352 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:26:33.214319 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c3f9c0_c03a_4dbf_a1d6_584000bed761.slice/crio-9427af80ab746077641d8a0ca0e6a258a72ca8604621bc447a62ed3f773224ae WatchSource:0}: Error finding container 9427af80ab746077641d8a0ca0e6a258a72ca8604621bc447a62ed3f773224ae: Status 404 returned error can't find the container with id 9427af80ab746077641d8a0ca0e6a258a72ca8604621bc447a62ed3f773224ae Sep 18 17:26:33.297925 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:33.297886 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" event={"ID":"21c3f9c0-c03a-4dbf-a1d6-584000bed761","Type":"ContainerStarted","Data":"3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef"} Sep 18 17:26:33.298063 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:33.297963 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" event={"ID":"21c3f9c0-c03a-4dbf-a1d6-584000bed761","Type":"ContainerStarted","Data":"9427af80ab746077641d8a0ca0e6a258a72ca8604621bc447a62ed3f773224ae"} Sep 18 17:26:37.318665 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:37.318579 2874 generic.go:356] "Generic (PLEG): container finished" podID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerID="3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef" exitCode=0 Sep 18 17:26:37.319009 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:37.318661 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" event={"ID":"21c3f9c0-c03a-4dbf-a1d6-584000bed761","Type":"ContainerDied","Data":"3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef"} Sep 18 17:26:37.319850 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:37.319835 2874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:26:52.406995 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:52.406959 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" event={"ID":"21c3f9c0-c03a-4dbf-a1d6-584000bed761","Type":"ContainerStarted","Data":"446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea"} Sep 18 17:26:52.407422 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:52.407019 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" event={"ID":"21c3f9c0-c03a-4dbf-a1d6-584000bed761","Type":"ContainerStarted","Data":"aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910"} Sep 18 17:26:52.407422 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:52.407294 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:52.407543 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:52.407430 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:52.408936 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:52.408911 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 18 17:26:52.439149 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:52.439104 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podStartSLOduration=2.45937954 podStartE2EDuration="20.439090933s" totalImagesPullingTime="14.210387636s" totalInitContainerRuntime="3.769323757s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:52.43726074 +0000 UTC m=+1985.771625458" watchObservedRunningTime="2026-09-18 17:26:52.439090933 +0000 UTC m=+1985.773455641" Sep 18 17:26:53.411677 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:53.411638 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 18 17:26:58.416496 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:58.416466 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:26:58.417010 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:26:58.416955 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 18 17:27:08.417912 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:27:08.417875 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 18 17:27:18.417902 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:27:18.417821 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 18 17:27:28.417415 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:27:28.417362 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 18 17:27:38.417818 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:27:38.417780 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 18 17:27:48.417737 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:27:48.417697 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 18 17:27:58.417385 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:27:58.417342 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 18 17:28:08.051221 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:08.051177 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl"] Sep 18 17:28:08.051744 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:08.051589 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" containerID="cri-o://aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910" gracePeriod=30 Sep 18 17:28:08.051744 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:08.051594 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kube-rbac-proxy" containerID="cri-o://446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea" gracePeriod=30 Sep 18 17:28:08.051872 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:08.051791 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Sep 18 17:28:08.412183 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:08.412141 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused" Sep 18 17:28:08.762555 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:08.762465 2874 generic.go:356] "Generic (PLEG): container finished" podID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerID="446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea" exitCode=2 Sep 18 17:28:08.762555 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:08.762534 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" event={"ID":"21c3f9c0-c03a-4dbf-a1d6-584000bed761","Type":"ContainerDied","Data":"446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea"} Sep 18 17:28:13.304589 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.304557 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:28:13.314994 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.314971 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-cabundle-cert\") pod \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " Sep 18 17:28:13.315122 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.315023 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kserve-provision-location\") pod \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " Sep 18 17:28:13.315189 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.315139 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdztq\" (UniqueName: \"kubernetes.io/projected/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-api-access-qdztq\") pod \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " Sep 18 17:28:13.315254 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.315210 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-rbac-proxy-sar-config\") pod \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " Sep 18 17:28:13.315254 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.315242 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3f9c0-c03a-4dbf-a1d6-584000bed761-proxy-tls\") pod \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\" (UID: \"21c3f9c0-c03a-4dbf-a1d6-584000bed761\") " Sep 18 17:28:13.315349 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.315274 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21c3f9c0-c03a-4dbf-a1d6-584000bed761" (UID: "21c3f9c0-c03a-4dbf-a1d6-584000bed761"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:28:13.315349 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.315289 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21c3f9c0-c03a-4dbf-a1d6-584000bed761" (UID: "21c3f9c0-c03a-4dbf-a1d6-584000bed761"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:28:13.315560 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.315543 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:28:13.315608 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.315567 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:28:13.315649 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.315602 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21c3f9c0-c03a-4dbf-a1d6-584000bed761" (UID: "21c3f9c0-c03a-4dbf-a1d6-584000bed761"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:28:13.317418 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.317369 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c3f9c0-c03a-4dbf-a1d6-584000bed761-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21c3f9c0-c03a-4dbf-a1d6-584000bed761" (UID: "21c3f9c0-c03a-4dbf-a1d6-584000bed761"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:28:13.317519 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.317472 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-api-access-qdztq" (OuterVolumeSpecName: "kube-api-access-qdztq") pod "21c3f9c0-c03a-4dbf-a1d6-584000bed761" (UID: "21c3f9c0-c03a-4dbf-a1d6-584000bed761"). InnerVolumeSpecName "kube-api-access-qdztq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:28:13.416371 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.416347 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdztq\" (UniqueName: \"kubernetes.io/projected/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-api-access-qdztq\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:28:13.416371 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.416371 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c3f9c0-c03a-4dbf-a1d6-584000bed761-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:28:13.416577 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.416381 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3f9c0-c03a-4dbf-a1d6-584000bed761-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:28:13.793055 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.792957 2874 generic.go:356] "Generic (PLEG): container finished" podID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerID="aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910" exitCode=139 Sep 18 17:28:13.793055 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.793044 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" Sep 18 17:28:13.793243 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.793037 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" event={"ID":"21c3f9c0-c03a-4dbf-a1d6-584000bed761","Type":"ContainerDied","Data":"aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910"} Sep 18 17:28:13.793243 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.793194 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl" event={"ID":"21c3f9c0-c03a-4dbf-a1d6-584000bed761","Type":"ContainerDied","Data":"9427af80ab746077641d8a0ca0e6a258a72ca8604621bc447a62ed3f773224ae"} Sep 18 17:28:13.793243 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.793220 2874 scope.go:117] "RemoveContainer" containerID="446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea" Sep 18 17:28:13.802904 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.802889 2874 scope.go:117] "RemoveContainer" containerID="aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910" Sep 18 17:28:13.811028 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.811007 2874 scope.go:117] "RemoveContainer" containerID="3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef" Sep 18 17:28:13.817447 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.817427 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl"] Sep 18 17:28:13.818548 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.818533 2874 scope.go:117] "RemoveContainer" containerID="446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea" Sep 18 17:28:13.818811 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:28:13.818789 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea\": container with ID starting with 446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea not found: ID does not exist" containerID="446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea" Sep 18 17:28:13.818870 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.818820 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea"} err="failed to get container status \"446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea\": rpc error: code = NotFound desc = could not find container \"446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea\": container with ID starting with 446e3618b47a788ecb8a8f9738491f9f0004595d8e83ddbe2c640d16470306ea not found: ID does not exist" Sep 18 17:28:13.818870 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.818840 2874 scope.go:117] "RemoveContainer" containerID="aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910" Sep 18 17:28:13.819136 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:28:13.819117 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910\": container with ID starting with aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910 not found: ID does not exist" containerID="aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910" Sep 18 17:28:13.819189 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.819140 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910"} err="failed to get container status \"aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910\": rpc error: code = NotFound desc = could not find container \"aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910\": container with ID starting with aad090595b0bc1687cf6db87757c39932ff8dd94a51dad1139e5de25f827c910 not found: ID does not exist" Sep 18 17:28:13.819189 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.819156 2874 scope.go:117] "RemoveContainer" containerID="3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef" Sep 18 17:28:13.819380 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:28:13.819362 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef\": container with ID starting with 3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef not found: ID does not exist" containerID="3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef" Sep 18 17:28:13.819668 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.819387 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef"} err="failed to get container status \"3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef\": rpc error: code = NotFound desc = could not find container \"3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef\": container with ID starting with 3a912a7c0c25ea0cbeea71254d0d68b0eac28eca39f87f5103ee4fffd5915bef not found: ID does not exist" Sep 18 17:28:13.822044 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:13.822024 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-8qfbl"] Sep 18 17:28:14.749259 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:14.749221 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" path="/var/lib/kubelet/pods/21c3f9c0-c03a-4dbf-a1d6-584000bed761/volumes" Sep 18 17:28:18.050162 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.050126 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9"] Sep 18 17:28:18.050572 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.050485 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kube-rbac-proxy" Sep 18 17:28:18.050572 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.050497 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kube-rbac-proxy" Sep 18 17:28:18.050572 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.050507 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" Sep 18 17:28:18.050572 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.050513 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" Sep 18 17:28:18.050572 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.050533 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="storage-initializer" Sep 18 17:28:18.050572 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.050538 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="storage-initializer" Sep 18 17:28:18.050779 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.050597 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kube-rbac-proxy" Sep 18 17:28:18.050779 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.050605 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c3f9c0-c03a-4dbf-a1d6-584000bed761" containerName="kserve-container" Sep 18 17:28:18.052732 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.052715 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.058917 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.058893 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 18 17:28:18.059085 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.058945 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Sep 18 17:28:18.059085 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.058955 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 18 17:28:18.060747 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.060720 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 18 17:28:18.063727 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.063707 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Sep 18 17:28:18.063800 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.063746 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 18 17:28:18.064798 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.064785 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\"" Sep 18 17:28:18.087340 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.087310 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9"] Sep 18 17:28:18.100203 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.100182 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc"] Sep 18 17:28:18.102573 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.102559 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.130793 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.130773 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\"" Sep 18 17:28:18.141002 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.140914 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc"] Sep 18 17:28:18.153702 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.153680 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.153806 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.153709 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfmn5\" (UniqueName: \"kubernetes.io/projected/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-api-access-zfmn5\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.153806 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.153735 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.153806 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.153772 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.153806 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.153801 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/023a6fc1-8da4-489a-892c-aa8131ff6c00-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.154013 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.153825 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54bv7\" (UniqueName: \"kubernetes.io/projected/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-api-access-54bv7\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.154013 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.153870 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.154013 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.153945 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.154013 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.153984 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.154185 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.154014 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/023a6fc1-8da4-489a-892c-aa8131ff6c00-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.254822 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.254786 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.254822 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.254827 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/023a6fc1-8da4-489a-892c-aa8131ff6c00-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.255087 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.254901 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54bv7\" (UniqueName: \"kubernetes.io/projected/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-api-access-54bv7\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.255087 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:28:18.254940 2874 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found Sep 18 17:28:18.255087 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.254962 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.255087 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:28:18.254998 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023a6fc1-8da4-489a-892c-aa8131ff6c00-proxy-tls podName:023a6fc1-8da4-489a-892c-aa8131ff6c00 nodeName:}" failed. No retries permitted until 2026-09-18 17:28:18.754979252 +0000 UTC m=+2072.089343951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/023a6fc1-8da4-489a-892c-aa8131ff6c00-proxy-tls") pod "isvc-canary-promote-predictor-5f4f848c44-fklqc" (UID: "023a6fc1-8da4-489a-892c-aa8131ff6c00") : secret "isvc-canary-promote-predictor-serving-cert" not found Sep 18 17:28:18.255087 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.255047 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.255087 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.255083 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.255452 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.255212 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/023a6fc1-8da4-489a-892c-aa8131ff6c00-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.255452 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.255303 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.255452 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.255330 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfmn5\" (UniqueName: \"kubernetes.io/projected/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-api-access-zfmn5\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.255452 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.255362 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.255452 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.255372 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.255719 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.255681 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/023a6fc1-8da4-489a-892c-aa8131ff6c00-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.255837 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.255811 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.255969 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.255811 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.256061 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.256043 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.256118 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.256063 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.257732 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.257713 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.288106 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.288073 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfmn5\" (UniqueName: \"kubernetes.io/projected/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-api-access-zfmn5\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.289855 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.289831 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54bv7\" (UniqueName: \"kubernetes.io/projected/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-api-access-54bv7\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.362859 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.362776 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:18.643522 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.643431 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9"] Sep 18 17:28:18.646178 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:28:18.646150 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272f7a4f_43ce_4a9c_8e0b_0d87e5a01f4b.slice/crio-bcd1dae1b2506e8a49bc1fe90500ebfe346a17bdc8e008d067e8aa65bf8cbab5 WatchSource:0}: Error finding container bcd1dae1b2506e8a49bc1fe90500ebfe346a17bdc8e008d067e8aa65bf8cbab5: Status 404 returned error can't find the container with id bcd1dae1b2506e8a49bc1fe90500ebfe346a17bdc8e008d067e8aa65bf8cbab5 Sep 18 17:28:18.760575 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.760542 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/023a6fc1-8da4-489a-892c-aa8131ff6c00-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.763076 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.763057 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/023a6fc1-8da4-489a-892c-aa8131ff6c00-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-fklqc\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:18.818081 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.818044 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" event={"ID":"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b","Type":"ContainerStarted","Data":"91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50"} Sep 18 17:28:18.818249 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:18.818104 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" event={"ID":"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b","Type":"ContainerStarted","Data":"bcd1dae1b2506e8a49bc1fe90500ebfe346a17bdc8e008d067e8aa65bf8cbab5"} Sep 18 17:28:19.012562 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:19.012474 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:19.303657 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:19.303627 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc"] Sep 18 17:28:19.306326 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:28:19.306292 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023a6fc1_8da4_489a_892c_aa8131ff6c00.slice/crio-a6424e403e97fd6a383d367afdbe2f965e995fa68e9df9fa5ae677021756fbf4 WatchSource:0}: Error finding container a6424e403e97fd6a383d367afdbe2f965e995fa68e9df9fa5ae677021756fbf4: Status 404 returned error can't find the container with id a6424e403e97fd6a383d367afdbe2f965e995fa68e9df9fa5ae677021756fbf4 Sep 18 17:28:19.824297 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:19.824265 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" event={"ID":"023a6fc1-8da4-489a-892c-aa8131ff6c00","Type":"ContainerStarted","Data":"ab9b89b1b899989c3ac648309e9eb7872f54f55995c9e0e44ac10b3966e93c70"} Sep 18 17:28:19.824509 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:19.824333 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" event={"ID":"023a6fc1-8da4-489a-892c-aa8131ff6c00","Type":"ContainerStarted","Data":"a6424e403e97fd6a383d367afdbe2f965e995fa68e9df9fa5ae677021756fbf4"} Sep 18 17:28:22.841073 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:22.840992 2874 generic.go:356] "Generic (PLEG): container finished" podID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerID="91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50" exitCode=0 Sep 18 17:28:22.841441 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:22.841073 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" event={"ID":"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b","Type":"ContainerDied","Data":"91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50"} Sep 18 17:28:23.850886 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:23.850850 2874 generic.go:356] "Generic (PLEG): container finished" podID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerID="ab9b89b1b899989c3ac648309e9eb7872f54f55995c9e0e44ac10b3966e93c70" exitCode=0 Sep 18 17:28:23.851344 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:23.850930 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" event={"ID":"023a6fc1-8da4-489a-892c-aa8131ff6c00","Type":"ContainerDied","Data":"ab9b89b1b899989c3ac648309e9eb7872f54f55995c9e0e44ac10b3966e93c70"} Sep 18 17:28:23.853142 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:23.853117 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" event={"ID":"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b","Type":"ContainerStarted","Data":"54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5"} Sep 18 17:28:23.853268 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:23.853176 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" event={"ID":"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b","Type":"ContainerStarted","Data":"eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba"} Sep 18 17:28:23.853466 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:23.853433 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:23.853523 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:23.853472 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:23.855317 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:23.855288 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Sep 18 17:28:23.905539 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:23.905487 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podStartSLOduration=2.067864884 podStartE2EDuration="5.90547184s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.837606956s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:23.904231119 +0000 UTC m=+2077.238595827" watchObservedRunningTime="2026-09-18 17:28:23.90547184 +0000 UTC m=+2077.239836547" Sep 18 17:28:24.860423 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:24.860375 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" event={"ID":"023a6fc1-8da4-489a-892c-aa8131ff6c00","Type":"ContainerStarted","Data":"6ee97883452c120500a48a80813ccf65da95919863c832659f2e14427317c9bb"} Sep 18 17:28:24.860792 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:24.860460 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" event={"ID":"023a6fc1-8da4-489a-892c-aa8131ff6c00","Type":"ContainerStarted","Data":"1920fe81fb03fd2f3b97d730c5b8ba93dbebd1070d477204734c1977c9025214"} Sep 18 17:28:24.860875 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:24.860837 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:24.860938 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:24.860893 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:24.860938 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:24.860844 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Sep 18 17:28:24.862531 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:24.862509 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 18 17:28:24.885746 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:24.885704 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podStartSLOduration=2.894412107 podStartE2EDuration="6.885690171s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.991278064s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:24.883886975 +0000 UTC m=+2078.218251682" watchObservedRunningTime="2026-09-18 17:28:24.885690171 +0000 UTC m=+2078.220054878" Sep 18 17:28:25.865531 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:25.865496 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 18 17:28:29.866844 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:29.866815 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:28:29.867507 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:29.867473 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Sep 18 17:28:30.871109 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:30.871082 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:28:30.871585 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:30.871560 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 18 17:28:39.867774 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:39.867735 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Sep 18 17:28:40.871972 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:40.871937 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 18 17:28:46.945574 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:46.945497 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:28:46.946562 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:46.946536 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:28:49.867418 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:49.867369 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Sep 18 17:28:50.871645 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:50.871609 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 18 17:28:59.868162 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:28:59.868124 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Sep 18 17:29:00.871507 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:00.871473 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 18 17:29:09.867640 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:09.867603 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Sep 18 17:29:10.872421 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:10.872372 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 18 17:29:19.867875 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:19.867829 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Sep 18 17:29:20.871522 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:20.871480 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 18 17:29:29.868205 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:29.868161 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Sep 18 17:29:30.871535 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:30.871499 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 18 17:29:39.868587 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:39.868559 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:29:40.064656 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:40.064622 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc"] Sep 18 17:29:40.064925 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:40.064889 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" containerID="cri-o://1920fe81fb03fd2f3b97d730c5b8ba93dbebd1070d477204734c1977c9025214" gracePeriod=30 Sep 18 17:29:40.065063 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:40.064967 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kube-rbac-proxy" containerID="cri-o://6ee97883452c120500a48a80813ccf65da95919863c832659f2e14427317c9bb" gracePeriod=30 Sep 18 17:29:40.065190 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:40.065158 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Sep 18 17:29:40.205705 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:40.205679 2874 generic.go:356] "Generic (PLEG): container finished" podID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerID="6ee97883452c120500a48a80813ccf65da95919863c832659f2e14427317c9bb" exitCode=2 Sep 18 17:29:40.205842 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:40.205757 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" event={"ID":"023a6fc1-8da4-489a-892c-aa8131ff6c00","Type":"ContainerDied","Data":"6ee97883452c120500a48a80813ccf65da95919863c832659f2e14427317c9bb"} Sep 18 17:29:40.866312 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:40.866271 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused" Sep 18 17:29:45.865937 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:45.865882 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused" Sep 18 17:29:50.066596 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:50.066562 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:29:50.866459 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:50.866421 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused" Sep 18 17:29:50.866733 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:50.866554 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:29:55.865890 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:29:55.865848 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused" Sep 18 17:30:00.866284 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:00.866236 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused" Sep 18 17:30:05.866735 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:05.866692 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused" Sep 18 17:30:10.346613 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.346557 2874 generic.go:356] "Generic (PLEG): container finished" podID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerID="1920fe81fb03fd2f3b97d730c5b8ba93dbebd1070d477204734c1977c9025214" exitCode=137 Sep 18 17:30:10.347085 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.346617 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" event={"ID":"023a6fc1-8da4-489a-892c-aa8131ff6c00","Type":"ContainerDied","Data":"1920fe81fb03fd2f3b97d730c5b8ba93dbebd1070d477204734c1977c9025214"} Sep 18 17:30:10.446877 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.446853 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:30:10.490855 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.490829 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfmn5\" (UniqueName: \"kubernetes.io/projected/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-api-access-zfmn5\") pod \"023a6fc1-8da4-489a-892c-aa8131ff6c00\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " Sep 18 17:30:10.491007 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.490865 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-rbac-proxy-sar-config\") pod \"023a6fc1-8da4-489a-892c-aa8131ff6c00\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " Sep 18 17:30:10.491007 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.490909 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-cabundle-cert\") pod \"023a6fc1-8da4-489a-892c-aa8131ff6c00\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " Sep 18 17:30:10.491007 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.490961 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/023a6fc1-8da4-489a-892c-aa8131ff6c00-kserve-provision-location\") pod \"023a6fc1-8da4-489a-892c-aa8131ff6c00\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " Sep 18 17:30:10.491182 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.491012 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/023a6fc1-8da4-489a-892c-aa8131ff6c00-proxy-tls\") pod \"023a6fc1-8da4-489a-892c-aa8131ff6c00\" (UID: \"023a6fc1-8da4-489a-892c-aa8131ff6c00\") " Sep 18 17:30:10.491327 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.491300 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "023a6fc1-8da4-489a-892c-aa8131ff6c00" (UID: "023a6fc1-8da4-489a-892c-aa8131ff6c00"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:30:10.491327 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.491313 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023a6fc1-8da4-489a-892c-aa8131ff6c00-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "023a6fc1-8da4-489a-892c-aa8131ff6c00" (UID: "023a6fc1-8da4-489a-892c-aa8131ff6c00"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:30:10.491477 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.491298 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "023a6fc1-8da4-489a-892c-aa8131ff6c00" (UID: "023a6fc1-8da4-489a-892c-aa8131ff6c00"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:30:10.493164 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.493137 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-api-access-zfmn5" (OuterVolumeSpecName: "kube-api-access-zfmn5") pod "023a6fc1-8da4-489a-892c-aa8131ff6c00" (UID: "023a6fc1-8da4-489a-892c-aa8131ff6c00"). InnerVolumeSpecName "kube-api-access-zfmn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:30:10.493269 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.493180 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023a6fc1-8da4-489a-892c-aa8131ff6c00-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "023a6fc1-8da4-489a-892c-aa8131ff6c00" (UID: "023a6fc1-8da4-489a-892c-aa8131ff6c00"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:30:10.592600 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.592505 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/023a6fc1-8da4-489a-892c-aa8131ff6c00-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:30:10.592600 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.592537 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/023a6fc1-8da4-489a-892c-aa8131ff6c00-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:30:10.592600 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.592553 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfmn5\" (UniqueName: \"kubernetes.io/projected/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-api-access-zfmn5\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:30:10.592600 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.592564 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:30:10.592600 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:10.592576 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/023a6fc1-8da4-489a-892c-aa8131ff6c00-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:30:11.353760 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:11.353726 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" event={"ID":"023a6fc1-8da4-489a-892c-aa8131ff6c00","Type":"ContainerDied","Data":"a6424e403e97fd6a383d367afdbe2f965e995fa68e9df9fa5ae677021756fbf4"} Sep 18 17:30:11.353760 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:11.353753 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc" Sep 18 17:30:11.354248 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:11.353795 2874 scope.go:117] "RemoveContainer" containerID="6ee97883452c120500a48a80813ccf65da95919863c832659f2e14427317c9bb" Sep 18 17:30:11.369339 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:11.369316 2874 scope.go:117] "RemoveContainer" containerID="1920fe81fb03fd2f3b97d730c5b8ba93dbebd1070d477204734c1977c9025214" Sep 18 17:30:11.376970 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:11.376945 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc"] Sep 18 17:30:11.378366 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:11.378346 2874 scope.go:117] "RemoveContainer" containerID="ab9b89b1b899989c3ac648309e9eb7872f54f55995c9e0e44ac10b3966e93c70" Sep 18 17:30:11.383424 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:11.383385 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-fklqc"] Sep 18 17:30:12.753711 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:12.753679 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" path="/var/lib/kubelet/pods/023a6fc1-8da4-489a-892c-aa8131ff6c00/volumes" Sep 18 17:30:29.583476 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.583368 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8"] Sep 18 17:30:29.584070 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.583763 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" Sep 18 17:30:29.584070 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.583777 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" Sep 18 17:30:29.584070 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.583796 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kube-rbac-proxy" Sep 18 17:30:29.584070 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.583802 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kube-rbac-proxy" Sep 18 17:30:29.584070 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.583813 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="storage-initializer" Sep 18 17:30:29.584070 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.583819 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="storage-initializer" Sep 18 17:30:29.584070 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.583879 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kube-rbac-proxy" Sep 18 17:30:29.584070 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.583895 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="023a6fc1-8da4-489a-892c-aa8131ff6c00" containerName="kserve-container" Sep 18 17:30:29.587335 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.587317 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.595579 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.595558 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\"" Sep 18 17:30:29.595740 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.595593 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\"" Sep 18 17:30:29.624888 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.624859 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8"] Sep 18 17:30:29.660171 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.660135 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dqn\" (UniqueName: \"kubernetes.io/projected/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-api-access-j7dqn\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.660341 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.660200 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdf70ad7-9c90-4784-801b-be5c74982f21-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.660341 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.660245 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.660341 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.660279 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.660341 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.660306 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf70ad7-9c90-4784-801b-be5c74982f21-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.761639 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.761606 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dqn\" (UniqueName: \"kubernetes.io/projected/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-api-access-j7dqn\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.761813 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.761662 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdf70ad7-9c90-4784-801b-be5c74982f21-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.761813 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.761694 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.761813 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.761726 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.761813 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.761753 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf70ad7-9c90-4784-801b-be5c74982f21-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.762046 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:30:29.761854 2874 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found Sep 18 17:30:29.762046 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:30:29.761952 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf70ad7-9c90-4784-801b-be5c74982f21-proxy-tls podName:fdf70ad7-9c90-4784-801b-be5c74982f21 nodeName:}" failed. No retries permitted until 2026-09-18 17:30:30.2619301 +0000 UTC m=+2203.596294799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fdf70ad7-9c90-4784-801b-be5c74982f21-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-jhkg8" (UID: "fdf70ad7-9c90-4784-801b-be5c74982f21") : secret "isvc-tensorflow-predictor-serving-cert" not found Sep 18 17:30:29.762160 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.762142 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf70ad7-9c90-4784-801b-be5c74982f21-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.762338 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.762320 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.762454 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.762438 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:29.781540 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:29.781516 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dqn\" (UniqueName: \"kubernetes.io/projected/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-api-access-j7dqn\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:30.266422 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:30.266370 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdf70ad7-9c90-4784-801b-be5c74982f21-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:30.268894 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:30.268873 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdf70ad7-9c90-4784-801b-be5c74982f21-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-jhkg8\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:30.497060 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:30.497012 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:30.786148 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:30.786111 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8"] Sep 18 17:30:30.791357 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:30:30.791323 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf70ad7_9c90_4784_801b_be5c74982f21.slice/crio-03eb352b64155998a78e2d2379984c32686b6134b2dd13561f59af4bb3cfd9b1 WatchSource:0}: Error finding container 03eb352b64155998a78e2d2379984c32686b6134b2dd13561f59af4bb3cfd9b1: Status 404 returned error can't find the container with id 03eb352b64155998a78e2d2379984c32686b6134b2dd13561f59af4bb3cfd9b1 Sep 18 17:30:31.446578 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:31.446536 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" event={"ID":"fdf70ad7-9c90-4784-801b-be5c74982f21","Type":"ContainerStarted","Data":"b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7"} Sep 18 17:30:31.446735 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:31.446608 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" event={"ID":"fdf70ad7-9c90-4784-801b-be5c74982f21","Type":"ContainerStarted","Data":"03eb352b64155998a78e2d2379984c32686b6134b2dd13561f59af4bb3cfd9b1"} Sep 18 17:30:35.466858 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:35.466820 2874 generic.go:356] "Generic (PLEG): container finished" podID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerID="b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7" exitCode=0 Sep 18 17:30:35.467342 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:35.466902 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" event={"ID":"fdf70ad7-9c90-4784-801b-be5c74982f21","Type":"ContainerDied","Data":"b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7"} Sep 18 17:30:39.492438 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:39.492382 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" event={"ID":"fdf70ad7-9c90-4784-801b-be5c74982f21","Type":"ContainerStarted","Data":"c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50"} Sep 18 17:30:39.492820 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:39.492468 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" event={"ID":"fdf70ad7-9c90-4784-801b-be5c74982f21","Type":"ContainerStarted","Data":"5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248"} Sep 18 17:30:39.492820 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:39.492660 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:39.534105 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:39.534005 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podStartSLOduration=2.217244248 podStartE2EDuration="10.533986042s" totalImagesPullingTime="3.799775062s" totalInitContainerRuntime="4.516966732s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:39.53323024 +0000 UTC m=+2212.867594971" watchObservedRunningTime="2026-09-18 17:30:39.533986042 +0000 UTC m=+2212.868350750" Sep 18 17:30:40.497743 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:40.497715 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:40.499267 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:40.499237 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 18 17:30:41.501900 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:41.501860 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 18 17:30:46.506434 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:46.506384 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:30:46.506932 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:46.506906 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused" Sep 18 17:30:56.508136 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:30:56.508107 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:31:10.072807 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:10.072766 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8"] Sep 18 17:31:10.073485 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:10.073003 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kserve-container" containerID="cri-o://5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248" gracePeriod=30 Sep 18 17:31:10.073485 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:10.073046 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kube-rbac-proxy" containerID="cri-o://c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50" gracePeriod=30 Sep 18 17:31:10.644491 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:10.644454 2874 generic.go:356] "Generic (PLEG): container finished" podID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerID="c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50" exitCode=2 Sep 18 17:31:10.644661 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:10.644504 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" event={"ID":"fdf70ad7-9c90-4784-801b-be5c74982f21","Type":"ContainerDied","Data":"c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50"} Sep 18 17:31:11.502737 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:11.502691 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused" Sep 18 17:31:16.502903 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:16.502861 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused" Sep 18 17:31:21.502418 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:21.502360 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused" Sep 18 17:31:21.502827 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:21.502514 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:31:26.502522 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:26.502473 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused" Sep 18 17:31:31.502992 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:31.502946 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused" Sep 18 17:31:36.502024 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:36.501986 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused" Sep 18 17:31:38.042264 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:38.042228 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9"] Sep 18 17:31:38.042683 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:38.042537 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" containerID="cri-o://eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba" gracePeriod=30 Sep 18 17:31:38.042683 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:38.042573 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kube-rbac-proxy" containerID="cri-o://54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5" gracePeriod=30 Sep 18 17:31:38.775535 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:38.775497 2874 generic.go:356] "Generic (PLEG): container finished" podID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerID="54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5" exitCode=2 Sep 18 17:31:38.775690 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:38.775573 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" event={"ID":"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b","Type":"ContainerDied","Data":"54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5"} Sep 18 17:31:39.861723 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:39.861683 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused" Sep 18 17:31:39.868083 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:39.868058 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Sep 18 17:31:40.727792 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.727761 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:31:40.786182 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.786149 2874 generic.go:356] "Generic (PLEG): container finished" podID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerID="5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248" exitCode=137 Sep 18 17:31:40.786182 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.786184 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" event={"ID":"fdf70ad7-9c90-4784-801b-be5c74982f21","Type":"ContainerDied","Data":"5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248"} Sep 18 17:31:40.786369 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.786227 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" event={"ID":"fdf70ad7-9c90-4784-801b-be5c74982f21","Type":"ContainerDied","Data":"03eb352b64155998a78e2d2379984c32686b6134b2dd13561f59af4bb3cfd9b1"} Sep 18 17:31:40.786369 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.786240 2874 scope.go:117] "RemoveContainer" containerID="c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50" Sep 18 17:31:40.786369 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.786229 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8" Sep 18 17:31:40.794523 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.794495 2874 scope.go:117] "RemoveContainer" containerID="5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248" Sep 18 17:31:40.802093 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.802075 2874 scope.go:117] "RemoveContainer" containerID="b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7" Sep 18 17:31:40.809621 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.809605 2874 scope.go:117] "RemoveContainer" containerID="c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50" Sep 18 17:31:40.809901 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:31:40.809883 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50\": container with ID starting with c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50 not found: ID does not exist" containerID="c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50" Sep 18 17:31:40.809968 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.809909 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50"} err="failed to get container status \"c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50\": rpc error: code = NotFound desc = could not find container \"c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50\": container with ID starting with c366b2deaee9f3205aa29437a7afbe1e9d62770ed6e6bb8312c70bbdbd068b50 not found: ID does not exist" Sep 18 17:31:40.809968 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.809925 2874 scope.go:117] "RemoveContainer" containerID="5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248" Sep 18 17:31:40.810157 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:31:40.810141 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248\": container with ID starting with 5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248 not found: ID does not exist" containerID="5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248" Sep 18 17:31:40.810201 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.810161 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248"} err="failed to get container status \"5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248\": rpc error: code = NotFound desc = could not find container \"5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248\": container with ID starting with 5688e19b8401b3d9773733441ca86f9694f5d18c55f7c13b35b8463d8a686248 not found: ID does not exist" Sep 18 17:31:40.810201 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.810176 2874 scope.go:117] "RemoveContainer" containerID="b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7" Sep 18 17:31:40.810422 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:31:40.810387 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7\": container with ID starting with b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7 not found: ID does not exist" containerID="b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7" Sep 18 17:31:40.810481 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.810425 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7"} err="failed to get container status \"b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7\": rpc error: code = NotFound desc = could not find container \"b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7\": container with ID starting with b61a13ae05187db1fc9a36f532202add14847e1bf3aba7a667a52eb7bbd7c2a7 not found: ID does not exist" Sep 18 17:31:40.878848 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.878825 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdf70ad7-9c90-4784-801b-be5c74982f21-proxy-tls\") pod \"fdf70ad7-9c90-4784-801b-be5c74982f21\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " Sep 18 17:31:40.879162 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.878866 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf70ad7-9c90-4784-801b-be5c74982f21-kserve-provision-location\") pod \"fdf70ad7-9c90-4784-801b-be5c74982f21\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " Sep 18 17:31:40.879162 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.878894 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7dqn\" (UniqueName: \"kubernetes.io/projected/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-api-access-j7dqn\") pod \"fdf70ad7-9c90-4784-801b-be5c74982f21\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " Sep 18 17:31:40.879162 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.878950 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-cabundle-cert\") pod \"fdf70ad7-9c90-4784-801b-be5c74982f21\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " Sep 18 17:31:40.879162 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.879028 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-rbac-proxy-sar-config\") pod \"fdf70ad7-9c90-4784-801b-be5c74982f21\" (UID: \"fdf70ad7-9c90-4784-801b-be5c74982f21\") " Sep 18 17:31:40.879421 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.879372 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fdf70ad7-9c90-4784-801b-be5c74982f21" (UID: "fdf70ad7-9c90-4784-801b-be5c74982f21"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:31:40.879514 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.879493 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fdf70ad7-9c90-4784-801b-be5c74982f21" (UID: "fdf70ad7-9c90-4784-801b-be5c74982f21"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:31:40.880982 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.880960 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf70ad7-9c90-4784-801b-be5c74982f21-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fdf70ad7-9c90-4784-801b-be5c74982f21" (UID: "fdf70ad7-9c90-4784-801b-be5c74982f21"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:31:40.881381 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.881361 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-api-access-j7dqn" (OuterVolumeSpecName: "kube-api-access-j7dqn") pod "fdf70ad7-9c90-4784-801b-be5c74982f21" (UID: "fdf70ad7-9c90-4784-801b-be5c74982f21"). InnerVolumeSpecName "kube-api-access-j7dqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:31:40.890216 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.890193 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf70ad7-9c90-4784-801b-be5c74982f21-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdf70ad7-9c90-4784-801b-be5c74982f21" (UID: "fdf70ad7-9c90-4784-801b-be5c74982f21"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:31:40.980482 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.980435 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf70ad7-9c90-4784-801b-be5c74982f21-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:31:40.980482 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.980487 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7dqn\" (UniqueName: \"kubernetes.io/projected/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-api-access-j7dqn\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:31:40.980482 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.980497 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:31:40.980731 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.980507 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdf70ad7-9c90-4784-801b-be5c74982f21-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:31:40.980731 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:40.980517 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdf70ad7-9c90-4784-801b-be5c74982f21-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:31:41.114941 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:41.114908 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8"] Sep 18 17:31:41.119203 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:41.119180 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jhkg8"] Sep 18 17:31:42.749755 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.749717 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" path="/var/lib/kubelet/pods/fdf70ad7-9c90-4784-801b-be5c74982f21/volumes" Sep 18 17:31:42.787804 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.787780 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:31:42.794034 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.794015 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54bv7\" (UniqueName: \"kubernetes.io/projected/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-api-access-54bv7\") pod \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " Sep 18 17:31:42.794131 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.794054 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-rbac-proxy-sar-config\") pod \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " Sep 18 17:31:42.794131 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.794096 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-cabundle-cert\") pod \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " Sep 18 17:31:42.794236 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.794137 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kserve-provision-location\") pod \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " Sep 18 17:31:42.794236 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.794184 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-proxy-tls\") pod \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\" (UID: \"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b\") " Sep 18 17:31:42.794522 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.794485 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" (UID: "272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:31:42.794522 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.794510 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" (UID: "272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:31:42.794659 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.794487 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" (UID: "272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:31:42.796342 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.796321 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-api-access-54bv7" (OuterVolumeSpecName: "kube-api-access-54bv7") pod "272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" (UID: "272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b"). InnerVolumeSpecName "kube-api-access-54bv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:31:42.796513 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.796493 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" (UID: "272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:31:42.798557 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.798532 2874 generic.go:356] "Generic (PLEG): container finished" podID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerID="eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba" exitCode=0 Sep 18 17:31:42.798655 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.798589 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" Sep 18 17:31:42.798655 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.798617 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" event={"ID":"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b","Type":"ContainerDied","Data":"eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba"} Sep 18 17:31:42.798734 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.798682 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9" event={"ID":"272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b","Type":"ContainerDied","Data":"bcd1dae1b2506e8a49bc1fe90500ebfe346a17bdc8e008d067e8aa65bf8cbab5"} Sep 18 17:31:42.798734 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.798716 2874 scope.go:117] "RemoveContainer" containerID="54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5" Sep 18 17:31:42.812686 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.812664 2874 scope.go:117] "RemoveContainer" containerID="eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba" Sep 18 17:31:42.822563 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.822543 2874 scope.go:117] "RemoveContainer" containerID="91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50" Sep 18 17:31:42.824714 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.824692 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9"] Sep 18 17:31:42.827348 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.827325 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-dhsj9"] Sep 18 17:31:42.831257 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.831237 2874 scope.go:117] "RemoveContainer" containerID="54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5" Sep 18 17:31:42.831566 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:31:42.831544 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5\": container with ID starting with 54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5 not found: ID does not exist" containerID="54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5" Sep 18 17:31:42.831636 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.831574 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5"} err="failed to get container status \"54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5\": rpc error: code = NotFound desc = could not find container \"54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5\": container with ID starting with 54239e036f7d42d6391ed0c5e2661b60cbcb05a513a8a8e94f61cbfc270ea1c5 not found: ID does not exist" Sep 18 17:31:42.831636 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.831593 2874 scope.go:117] "RemoveContainer" containerID="eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba" Sep 18 17:31:42.831847 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:31:42.831834 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba\": container with ID starting with eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba not found: ID does not exist" containerID="eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba" Sep 18 17:31:42.831891 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.831852 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba"} err="failed to get container status \"eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba\": rpc error: code = NotFound desc = could not find container \"eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba\": container with ID starting with eda48eac2a2e766bfcd229514a7cd233e88ba0e663a1ec9330a8a1de16c02aba not found: ID does not exist" Sep 18 17:31:42.831891 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.831864 2874 scope.go:117] "RemoveContainer" containerID="91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50" Sep 18 17:31:42.832116 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:31:42.832102 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50\": container with ID starting with 91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50 not found: ID does not exist" containerID="91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50" Sep 18 17:31:42.832158 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.832122 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50"} err="failed to get container status \"91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50\": rpc error: code = NotFound desc = could not find container \"91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50\": container with ID starting with 91d821ca81e358584038edfc5b857c374a5640a71a32297f6618bed588161b50 not found: ID does not exist" Sep 18 17:31:42.895357 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.895293 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:31:42.895357 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.895318 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:31:42.895357 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.895328 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:31:42.895357 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.895338 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:31:42.895357 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:42.895350 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54bv7\" (UniqueName: \"kubernetes.io/projected/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b-kube-api-access-54bv7\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:31:43.226887 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.226810 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq"] Sep 18 17:31:43.227145 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227130 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kube-rbac-proxy" Sep 18 17:31:43.227209 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227148 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kube-rbac-proxy" Sep 18 17:31:43.227209 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227172 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="storage-initializer" Sep 18 17:31:43.227209 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227180 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="storage-initializer" Sep 18 17:31:43.227209 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227203 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kube-rbac-proxy" Sep 18 17:31:43.227209 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227211 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kube-rbac-proxy" Sep 18 17:31:43.227444 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227220 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="storage-initializer" Sep 18 17:31:43.227444 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227226 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="storage-initializer" Sep 18 17:31:43.227444 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227235 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" Sep 18 17:31:43.227444 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227240 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" Sep 18 17:31:43.227444 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227246 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kserve-container" Sep 18 17:31:43.227444 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227251 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kserve-container" Sep 18 17:31:43.227444 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227330 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kserve-container" Sep 18 17:31:43.227444 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227337 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kube-rbac-proxy" Sep 18 17:31:43.227444 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227346 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" containerName="kserve-container" Sep 18 17:31:43.227444 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.227352 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf70ad7-9c90-4784-801b-be5c74982f21" containerName="kube-rbac-proxy" Sep 18 17:31:43.230451 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.230395 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.234728 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.234707 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 18 17:31:43.234866 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.234843 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\"" Sep 18 17:31:43.234942 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.234876 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Sep 18 17:31:43.234942 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.234889 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 18 17:31:43.235052 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.234943 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 18 17:31:43.235170 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.235157 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 18 17:31:43.235657 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.235643 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\"" Sep 18 17:31:43.241971 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.241949 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq"] Sep 18 17:31:43.297629 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.297597 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.297772 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.297634 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.297772 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.297656 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdc8x\" (UniqueName: \"kubernetes.io/projected/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-api-access-pdc8x\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.297772 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.297744 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.297903 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.297789 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.317609 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.317582 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm"] Sep 18 17:31:43.320211 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.320194 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.322566 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.322544 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Sep 18 17:31:43.331412 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.331377 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm"] Sep 18 17:31:43.398928 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.398894 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.398928 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.398930 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.399122 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.398952 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.399122 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.398979 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.399122 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.399017 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.399122 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.399083 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.399295 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.399134 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdc8x\" (UniqueName: \"kubernetes.io/projected/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-api-access-pdc8x\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.399295 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.399156 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv845\" (UniqueName: \"kubernetes.io/projected/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-api-access-vv845\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.399295 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.399206 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.399295 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.399246 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.399561 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.399460 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.399730 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.399705 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.399929 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.399910 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.401825 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.401807 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.407648 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.407628 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdc8x\" (UniqueName: \"kubernetes.io/projected/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-api-access-pdc8x\") pod \"isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.500559 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.500471 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.500559 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.500520 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.500747 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.500567 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.500747 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.500590 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.500747 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:31:43.500706 2874 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found Sep 18 17:31:43.500747 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.500720 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv845\" (UniqueName: \"kubernetes.io/projected/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-api-access-vv845\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.500916 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:31:43.500778 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-proxy-tls podName:aefc9dfd-6251-41d2-a7a9-92e9ee84b149 nodeName:}" failed. No retries permitted until 2026-09-18 17:31:44.000758078 +0000 UTC m=+2277.335122783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-proxy-tls") pod "isvc-canary-rollback-predictor-69978998f6-5zzhm" (UID: "aefc9dfd-6251-41d2-a7a9-92e9ee84b149") : secret "isvc-canary-rollback-predictor-serving-cert" not found Sep 18 17:31:43.500980 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.500915 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.501178 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.501160 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.501298 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.501279 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.511972 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.511942 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv845\" (UniqueName: \"kubernetes.io/projected/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-api-access-vv845\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:43.542657 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.542625 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:43.814770 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:43.814738 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq"] Sep 18 17:31:43.816943 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:31:43.816915 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5194fa_ac1a_4319_9a1f_3e9062796b6a.slice/crio-dfe2984bc217d0a301d75e6d2ed1a7ee3c72c2d4cb3bf7d2e3b94845dae3e7f6 WatchSource:0}: Error finding container dfe2984bc217d0a301d75e6d2ed1a7ee3c72c2d4cb3bf7d2e3b94845dae3e7f6: Status 404 returned error can't find the container with id dfe2984bc217d0a301d75e6d2ed1a7ee3c72c2d4cb3bf7d2e3b94845dae3e7f6 Sep 18 17:31:44.005161 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:44.005133 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:44.007734 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:44.007705 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-5zzhm\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:44.230878 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:44.230843 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:44.506391 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:44.506363 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm"] Sep 18 17:31:44.507247 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:31:44.507224 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefc9dfd_6251_41d2_a7a9_92e9ee84b149.slice/crio-d008f1b656091ad10e6e5a2523ff0badb2e220194cb3b4a753b5ae7644612e43 WatchSource:0}: Error finding container d008f1b656091ad10e6e5a2523ff0badb2e220194cb3b4a753b5ae7644612e43: Status 404 returned error can't find the container with id d008f1b656091ad10e6e5a2523ff0badb2e220194cb3b4a753b5ae7644612e43 Sep 18 17:31:44.749121 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:44.749090 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b" path="/var/lib/kubelet/pods/272f7a4f-43ce-4a9c-8e0b-0d87e5a01f4b/volumes" Sep 18 17:31:44.809961 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:44.809885 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" event={"ID":"aefc9dfd-6251-41d2-a7a9-92e9ee84b149","Type":"ContainerStarted","Data":"392bf1b0a597f3af6003cf7b5ec209b2d85087a1b8025dbee5376183f80a7897"} Sep 18 17:31:44.809961 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:44.809944 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" event={"ID":"aefc9dfd-6251-41d2-a7a9-92e9ee84b149","Type":"ContainerStarted","Data":"d008f1b656091ad10e6e5a2523ff0badb2e220194cb3b4a753b5ae7644612e43"} Sep 18 17:31:44.811688 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:44.811660 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" event={"ID":"6e5194fa-ac1a-4319-9a1f-3e9062796b6a","Type":"ContainerStarted","Data":"bcecfb16a1d1c5dc367e962d85e7c286b8c313139816949ac117551ef27c9771"} Sep 18 17:31:44.811799 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:44.811710 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" event={"ID":"6e5194fa-ac1a-4319-9a1f-3e9062796b6a","Type":"ContainerStarted","Data":"dfe2984bc217d0a301d75e6d2ed1a7ee3c72c2d4cb3bf7d2e3b94845dae3e7f6"} Sep 18 17:31:47.828866 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:47.828838 2874 generic.go:356] "Generic (PLEG): container finished" podID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerID="bcecfb16a1d1c5dc367e962d85e7c286b8c313139816949ac117551ef27c9771" exitCode=0 Sep 18 17:31:47.829261 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:47.828907 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" event={"ID":"6e5194fa-ac1a-4319-9a1f-3e9062796b6a","Type":"ContainerDied","Data":"bcecfb16a1d1c5dc367e962d85e7c286b8c313139816949ac117551ef27c9771"} Sep 18 17:31:48.835129 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:48.835035 2874 generic.go:356] "Generic (PLEG): container finished" podID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerID="392bf1b0a597f3af6003cf7b5ec209b2d85087a1b8025dbee5376183f80a7897" exitCode=0 Sep 18 17:31:48.835129 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:48.835105 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" event={"ID":"aefc9dfd-6251-41d2-a7a9-92e9ee84b149","Type":"ContainerDied","Data":"392bf1b0a597f3af6003cf7b5ec209b2d85087a1b8025dbee5376183f80a7897"} Sep 18 17:31:48.837749 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:48.837728 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" event={"ID":"6e5194fa-ac1a-4319-9a1f-3e9062796b6a","Type":"ContainerStarted","Data":"c7bef983062fb39eef2a6b21866fa498b2037751c29298f5cb22bfdaacaeca5c"} Sep 18 17:31:48.837820 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:48.837786 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" event={"ID":"6e5194fa-ac1a-4319-9a1f-3e9062796b6a","Type":"ContainerStarted","Data":"fb865f70c6d427ee757f3d214b5c8d1c11739f896e5e52a110082ace70368e02"} Sep 18 17:31:48.838010 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:48.837994 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:48.877008 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:48.876967 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podStartSLOduration=1.9520743299999999 podStartE2EDuration="5.876948075s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.924873745s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:31:48.875717098 +0000 UTC m=+2282.210081808" watchObservedRunningTime="2026-09-18 17:31:48.876948075 +0000 UTC m=+2282.211312782" Sep 18 17:31:49.845238 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:49.845197 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" event={"ID":"aefc9dfd-6251-41d2-a7a9-92e9ee84b149","Type":"ContainerStarted","Data":"579c09bf2d8e8382e7f670925ebd7ad75c7554fe26a2ba140a4ce1caf7eb2e87"} Sep 18 17:31:49.845636 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:49.845266 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" event={"ID":"aefc9dfd-6251-41d2-a7a9-92e9ee84b149","Type":"ContainerStarted","Data":"d65e67b1b3bfe2167a90e0b823f1e202f4497ee094a9f290b3f4b1a3b6bf5403"} Sep 18 17:31:49.845636 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:49.845501 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:49.845763 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:49.845647 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:49.845763 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:49.845689 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:49.846912 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:49.846883 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:31:49.846912 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:49.846903 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:31:49.866370 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:49.866326 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podStartSLOduration=2.991649838 podStartE2EDuration="6.86631273s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.874662892s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:31:49.865141697 +0000 UTC m=+2283.199506587" watchObservedRunningTime="2026-09-18 17:31:49.86631273 +0000 UTC m=+2283.200677436" Sep 18 17:31:50.849772 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:50.849714 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:31:50.850197 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:50.849831 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:31:55.854456 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:55.854424 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:31:55.854851 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:55.854552 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:31:55.855065 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:55.855037 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:31:55.855347 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:31:55.855320 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:32:02.558047 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.558013 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg"] Sep 18 17:32:02.561646 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.561625 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.563850 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.563832 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\"" Sep 18 17:32:02.564129 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.564116 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\"" Sep 18 17:32:02.573807 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.573786 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg"] Sep 18 17:32:02.671610 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.671569 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85dbd611-d09e-4113-9bd2-530a7c919194-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.671788 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.671641 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85dbd611-d09e-4113-9bd2-530a7c919194-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.671788 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.671746 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.671788 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.671785 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxth2\" (UniqueName: \"kubernetes.io/projected/85dbd611-d09e-4113-9bd2-530a7c919194-kube-api-access-gxth2\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.671897 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.671802 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.772240 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.772204 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85dbd611-d09e-4113-9bd2-530a7c919194-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.772459 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.772264 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.772459 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.772281 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxth2\" (UniqueName: \"kubernetes.io/projected/85dbd611-d09e-4113-9bd2-530a7c919194-kube-api-access-gxth2\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.772459 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.772296 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.772459 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.772318 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85dbd611-d09e-4113-9bd2-530a7c919194-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.772709 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:32:02.772478 2874 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found Sep 18 17:32:02.772709 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:32:02.772538 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85dbd611-d09e-4113-9bd2-530a7c919194-proxy-tls podName:85dbd611-d09e-4113-9bd2-530a7c919194 nodeName:}" failed. No retries permitted until 2026-09-18 17:32:03.272518302 +0000 UTC m=+2296.606882993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/85dbd611-d09e-4113-9bd2-530a7c919194-proxy-tls") pod "isvc-triton-predictor-5d7db58676-md2cg" (UID: "85dbd611-d09e-4113-9bd2-530a7c919194") : secret "isvc-triton-predictor-serving-cert" not found Sep 18 17:32:02.772709 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.772661 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85dbd611-d09e-4113-9bd2-530a7c919194-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.773035 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.773014 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.773121 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.773049 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:02.781263 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:02.781240 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxth2\" (UniqueName: \"kubernetes.io/projected/85dbd611-d09e-4113-9bd2-530a7c919194-kube-api-access-gxth2\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:03.277109 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:03.277070 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85dbd611-d09e-4113-9bd2-530a7c919194-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:03.279695 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:03.279671 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85dbd611-d09e-4113-9bd2-530a7c919194-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-md2cg\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:03.472787 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:03.472749 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:32:03.749533 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:03.749509 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg"] Sep 18 17:32:03.751987 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:32:03.751958 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dbd611_d09e_4113_9bd2_530a7c919194.slice/crio-623046f071429455ff099d946b6485055f7be0e59a93ac0a2f573e63b7d524a2 WatchSource:0}: Error finding container 623046f071429455ff099d946b6485055f7be0e59a93ac0a2f573e63b7d524a2: Status 404 returned error can't find the container with id 623046f071429455ff099d946b6485055f7be0e59a93ac0a2f573e63b7d524a2 Sep 18 17:32:03.915875 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:03.915838 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" event={"ID":"85dbd611-d09e-4113-9bd2-530a7c919194","Type":"ContainerStarted","Data":"eb1a47d6c9da3164eb22508c56bbc24221008ed7e3e81970c77b4bf97d18891b"} Sep 18 17:32:03.916072 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:03.915895 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" event={"ID":"85dbd611-d09e-4113-9bd2-530a7c919194","Type":"ContainerStarted","Data":"623046f071429455ff099d946b6485055f7be0e59a93ac0a2f573e63b7d524a2"} Sep 18 17:32:05.855278 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:05.855240 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:32:05.855700 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:05.855242 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:32:07.937279 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:07.937243 2874 generic.go:356] "Generic (PLEG): container finished" podID="85dbd611-d09e-4113-9bd2-530a7c919194" containerID="eb1a47d6c9da3164eb22508c56bbc24221008ed7e3e81970c77b4bf97d18891b" exitCode=0 Sep 18 17:32:07.937718 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:07.937325 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" event={"ID":"85dbd611-d09e-4113-9bd2-530a7c919194","Type":"ContainerDied","Data":"eb1a47d6c9da3164eb22508c56bbc24221008ed7e3e81970c77b4bf97d18891b"} Sep 18 17:32:07.938596 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:07.938578 2874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:32:15.857508 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:15.856649 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:32:15.858393 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:15.857875 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:32:25.855364 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:25.855320 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:32:25.855979 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:25.855801 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:32:35.855446 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:35.855064 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:32:35.855446 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:35.855274 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:32:45.855305 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:45.855247 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:32:45.855914 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:45.855251 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:32:55.855464 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:55.855188 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:32:55.855464 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:32:55.855208 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:33:02.743966 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:02.743899 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:33:05.855127 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:05.855080 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:33:12.743979 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:12.743924 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:33:13.743605 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:13.743567 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:33:22.744485 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:22.744382 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:33:23.744397 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:23.744340 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:33:32.751310 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:32.751278 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:33:33.744427 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:33.744376 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:33:33.978961 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:33.978920 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm"] Sep 18 17:33:33.979438 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:33.979180 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" containerID="cri-o://d65e67b1b3bfe2167a90e0b823f1e202f4497ee094a9f290b3f4b1a3b6bf5403" gracePeriod=30 Sep 18 17:33:33.979438 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:33.979214 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kube-rbac-proxy" containerID="cri-o://579c09bf2d8e8382e7f670925ebd7ad75c7554fe26a2ba140a4ce1caf7eb2e87" gracePeriod=30 Sep 18 17:33:34.514717 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:34.514688 2874 generic.go:356] "Generic (PLEG): container finished" podID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerID="579c09bf2d8e8382e7f670925ebd7ad75c7554fe26a2ba140a4ce1caf7eb2e87" exitCode=2 Sep 18 17:33:34.514870 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:34.514767 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" event={"ID":"aefc9dfd-6251-41d2-a7a9-92e9ee84b149","Type":"ContainerDied","Data":"579c09bf2d8e8382e7f670925ebd7ad75c7554fe26a2ba140a4ce1caf7eb2e87"} Sep 18 17:33:35.850176 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:35.850127 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused" Sep 18 17:33:38.331324 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.331283 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq"] Sep 18 17:33:38.331935 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.331594 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" containerID="cri-o://fb865f70c6d427ee757f3d214b5c8d1c11739f896e5e52a110082ace70368e02" gracePeriod=30 Sep 18 17:33:38.331935 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.331725 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kube-rbac-proxy" containerID="cri-o://c7bef983062fb39eef2a6b21866fa498b2037751c29298f5cb22bfdaacaeca5c" gracePeriod=30 Sep 18 17:33:38.425516 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.425480 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns"] Sep 18 17:33:38.455495 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.455460 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns"] Sep 18 17:33:38.455692 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.455567 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.617103 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.617003 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6srk9\" (UniqueName: \"kubernetes.io/projected/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-api-access-6srk9\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.617103 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.617080 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.617343 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.617124 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.617343 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.617167 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.617343 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.617188 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.718475 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.718431 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.718680 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.718499 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.718680 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.718527 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.718680 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.718621 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6srk9\" (UniqueName: \"kubernetes.io/projected/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-api-access-6srk9\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.718680 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.718661 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.718980 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.718956 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.719299 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.719267 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.719299 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.719289 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.721570 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.721544 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.728077 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.728052 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6srk9\" (UniqueName: \"kubernetes.io/projected/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-api-access-6srk9\") pod \"isvc-canary-rollback-predictor-69978998f6-4qfns\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:38.768692 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:38.768657 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:33:40.850941 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:40.850900 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused" Sep 18 17:33:40.850941 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:40.850899 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused" Sep 18 17:33:42.743976 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:42.743936 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:33:43.743697 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:43.743656 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:33:45.849960 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:45.849918 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused" Sep 18 17:33:45.850562 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:45.849914 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused" Sep 18 17:33:45.850562 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:45.850164 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:33:50.850661 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:50.850602 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused" Sep 18 17:33:50.851270 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:50.850603 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused" Sep 18 17:33:50.851270 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:50.850855 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:33:52.744608 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:52.744566 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:33:53.744256 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:53.744210 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:33:55.850201 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:55.850153 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused" Sep 18 17:33:55.850698 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:33:55.850169 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused" Sep 18 17:34:00.850115 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:00.850070 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused" Sep 18 17:34:00.850672 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:00.850074 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused" Sep 18 17:34:02.744605 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:02.744561 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused" Sep 18 17:34:02.751055 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:02.750992 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:34:03.678566 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.678529 2874 generic.go:356] "Generic (PLEG): container finished" podID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerID="d65e67b1b3bfe2167a90e0b823f1e202f4497ee094a9f290b3f4b1a3b6bf5403" exitCode=0 Sep 18 17:34:03.678729 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.678599 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" event={"ID":"aefc9dfd-6251-41d2-a7a9-92e9ee84b149","Type":"ContainerDied","Data":"d65e67b1b3bfe2167a90e0b823f1e202f4497ee094a9f290b3f4b1a3b6bf5403"} Sep 18 17:34:03.681235 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.681213 2874 generic.go:356] "Generic (PLEG): container finished" podID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerID="c7bef983062fb39eef2a6b21866fa498b2037751c29298f5cb22bfdaacaeca5c" exitCode=2 Sep 18 17:34:03.681235 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.681232 2874 generic.go:356] "Generic (PLEG): container finished" podID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerID="fb865f70c6d427ee757f3d214b5c8d1c11739f896e5e52a110082ace70368e02" exitCode=0 Sep 18 17:34:03.681427 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.681278 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" event={"ID":"6e5194fa-ac1a-4319-9a1f-3e9062796b6a","Type":"ContainerDied","Data":"c7bef983062fb39eef2a6b21866fa498b2037751c29298f5cb22bfdaacaeca5c"} Sep 18 17:34:03.681427 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.681345 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" event={"ID":"6e5194fa-ac1a-4319-9a1f-3e9062796b6a","Type":"ContainerDied","Data":"fb865f70c6d427ee757f3d214b5c8d1c11739f896e5e52a110082ace70368e02"} Sep 18 17:34:03.743478 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.743443 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused" Sep 18 17:34:03.743604 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.743568 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:34:03.939561 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.939515 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:34:03.941483 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.941461 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:34:03.973091 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.973062 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:34:03.983920 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:03.983894 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:34:04.051812 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.051775 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv845\" (UniqueName: \"kubernetes.io/projected/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-api-access-vv845\") pod \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " Sep 18 17:34:04.051994 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.051825 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kserve-provision-location\") pod \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " Sep 18 17:34:04.051994 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.051862 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdc8x\" (UniqueName: \"kubernetes.io/projected/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-api-access-pdc8x\") pod \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " Sep 18 17:34:04.051994 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.051937 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-rbac-proxy-sar-config\") pod \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " Sep 18 17:34:04.052177 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.051999 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-proxy-tls\") pod \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " Sep 18 17:34:04.052177 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052018 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-cabundle-cert\") pod \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\" (UID: \"6e5194fa-ac1a-4319-9a1f-3e9062796b6a\") " Sep 18 17:34:04.052177 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052065 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-rbac-proxy-sar-config\") pod \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " Sep 18 17:34:04.052177 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052157 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kserve-provision-location\") pod \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " Sep 18 17:34:04.052177 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052163 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e5194fa-ac1a-4319-9a1f-3e9062796b6a" (UID: "6e5194fa-ac1a-4319-9a1f-3e9062796b6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:04.052455 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052183 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-proxy-tls\") pod \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " Sep 18 17:34:04.052455 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052247 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-cabundle-cert\") pod \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\" (UID: \"aefc9dfd-6251-41d2-a7a9-92e9ee84b149\") " Sep 18 17:34:04.052455 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052227 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e5194fa-ac1a-4319-9a1f-3e9062796b6a" (UID: "6e5194fa-ac1a-4319-9a1f-3e9062796b6a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:34:04.052605 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052558 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:04.052605 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052564 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aefc9dfd-6251-41d2-a7a9-92e9ee84b149" (UID: "aefc9dfd-6251-41d2-a7a9-92e9ee84b149"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:04.052605 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052580 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:04.052754 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052604 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aefc9dfd-6251-41d2-a7a9-92e9ee84b149" (UID: "aefc9dfd-6251-41d2-a7a9-92e9ee84b149"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:34:04.052808 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052686 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aefc9dfd-6251-41d2-a7a9-92e9ee84b149" (UID: "aefc9dfd-6251-41d2-a7a9-92e9ee84b149"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:34:04.052808 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.052767 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e5194fa-ac1a-4319-9a1f-3e9062796b6a" (UID: "6e5194fa-ac1a-4319-9a1f-3e9062796b6a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:34:04.054760 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.054733 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aefc9dfd-6251-41d2-a7a9-92e9ee84b149" (UID: "aefc9dfd-6251-41d2-a7a9-92e9ee84b149"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:34:04.054858 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.054783 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-api-access-vv845" (OuterVolumeSpecName: "kube-api-access-vv845") pod "aefc9dfd-6251-41d2-a7a9-92e9ee84b149" (UID: "aefc9dfd-6251-41d2-a7a9-92e9ee84b149"). InnerVolumeSpecName "kube-api-access-vv845". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:34:04.054858 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.054828 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-api-access-pdc8x" (OuterVolumeSpecName: "kube-api-access-pdc8x") pod "6e5194fa-ac1a-4319-9a1f-3e9062796b6a" (UID: "6e5194fa-ac1a-4319-9a1f-3e9062796b6a"). InnerVolumeSpecName "kube-api-access-pdc8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:34:04.054858 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.054824 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e5194fa-ac1a-4319-9a1f-3e9062796b6a" (UID: "6e5194fa-ac1a-4319-9a1f-3e9062796b6a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:34:04.153656 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.153624 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:04.153656 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.153653 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv845\" (UniqueName: \"kubernetes.io/projected/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-api-access-vv845\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:04.153855 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.153663 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdc8x\" (UniqueName: \"kubernetes.io/projected/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-kube-api-access-pdc8x\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:04.153855 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.153674 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:04.153855 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.153684 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e5194fa-ac1a-4319-9a1f-3e9062796b6a-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:04.153855 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.153691 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:04.153855 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.153699 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:04.153855 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.153706 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aefc9dfd-6251-41d2-a7a9-92e9ee84b149-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:04.188168 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.188141 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns"] Sep 18 17:34:04.190447 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:34:04.190376 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87d87f9_4fda_46a8_9efd_ac38b0c0d695.slice/crio-20c78b9ea650661deeba28f997f1b93e54ea77f4598b3b921e9fee92eadba7d6 WatchSource:0}: Error finding container 20c78b9ea650661deeba28f997f1b93e54ea77f4598b3b921e9fee92eadba7d6: Status 404 returned error can't find the container with id 20c78b9ea650661deeba28f997f1b93e54ea77f4598b3b921e9fee92eadba7d6 Sep 18 17:34:04.688361 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.688260 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" event={"ID":"b87d87f9-4fda-46a8-9efd-ac38b0c0d695","Type":"ContainerStarted","Data":"5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9"} Sep 18 17:34:04.688361 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.688350 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" event={"ID":"b87d87f9-4fda-46a8-9efd-ac38b0c0d695","Type":"ContainerStarted","Data":"20c78b9ea650661deeba28f997f1b93e54ea77f4598b3b921e9fee92eadba7d6"} Sep 18 17:34:04.690534 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.690507 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" Sep 18 17:34:04.690680 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.690507 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm" event={"ID":"aefc9dfd-6251-41d2-a7a9-92e9ee84b149","Type":"ContainerDied","Data":"d008f1b656091ad10e6e5a2523ff0badb2e220194cb3b4a753b5ae7644612e43"} Sep 18 17:34:04.690680 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.690657 2874 scope.go:117] "RemoveContainer" containerID="579c09bf2d8e8382e7f670925ebd7ad75c7554fe26a2ba140a4ce1caf7eb2e87" Sep 18 17:34:04.693131 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.693105 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" Sep 18 17:34:04.693131 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.693121 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq" event={"ID":"6e5194fa-ac1a-4319-9a1f-3e9062796b6a","Type":"ContainerDied","Data":"dfe2984bc217d0a301d75e6d2ed1a7ee3c72c2d4cb3bf7d2e3b94845dae3e7f6"} Sep 18 17:34:04.700984 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.700965 2874 scope.go:117] "RemoveContainer" containerID="d65e67b1b3bfe2167a90e0b823f1e202f4497ee094a9f290b3f4b1a3b6bf5403" Sep 18 17:34:04.712169 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.712140 2874 scope.go:117] "RemoveContainer" containerID="392bf1b0a597f3af6003cf7b5ec209b2d85087a1b8025dbee5376183f80a7897" Sep 18 17:34:04.721545 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.721518 2874 scope.go:117] "RemoveContainer" containerID="c7bef983062fb39eef2a6b21866fa498b2037751c29298f5cb22bfdaacaeca5c" Sep 18 17:34:04.722488 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.722466 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm"] Sep 18 17:34:04.727123 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.727099 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-5zzhm"] Sep 18 17:34:04.732445 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.732426 2874 scope.go:117] "RemoveContainer" containerID="fb865f70c6d427ee757f3d214b5c8d1c11739f896e5e52a110082ace70368e02" Sep 18 17:34:04.738118 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.738080 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq"] Sep 18 17:34:04.740780 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.740747 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-ddcc7cc7c-6kmlq"] Sep 18 17:34:04.742647 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.742626 2874 scope.go:117] "RemoveContainer" containerID="bcecfb16a1d1c5dc367e962d85e7c286b8c313139816949ac117551ef27c9771" Sep 18 17:34:04.750088 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.750067 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" path="/var/lib/kubelet/pods/6e5194fa-ac1a-4319-9a1f-3e9062796b6a/volumes" Sep 18 17:34:04.750635 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:04.750618 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" path="/var/lib/kubelet/pods/aefc9dfd-6251-41d2-a7a9-92e9ee84b149/volumes" Sep 18 17:34:05.701563 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:05.701527 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" event={"ID":"85dbd611-d09e-4113-9bd2-530a7c919194","Type":"ContainerStarted","Data":"f567540db9c0166436db81b32ae8b49c57c02e1c3876de74bbd9e20adbfdda0f"} Sep 18 17:34:05.701963 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:05.701585 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" event={"ID":"85dbd611-d09e-4113-9bd2-530a7c919194","Type":"ContainerStarted","Data":"16f23d9a71484247e0265357951d34a6fee7d177b438659ac58b59a1ea289995"} Sep 18 17:34:05.701963 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:05.701769 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:34:05.724183 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:05.724141 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" podStartSLOduration=2.864271247 podStartE2EDuration="2m3.724128323s" totalImagesPullingTime="1m56.818842962s" totalInitContainerRuntime="4.041014114s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:05.723319258 +0000 UTC m=+2419.057683967" watchObservedRunningTime="2026-09-18 17:34:05.724128323 +0000 UTC m=+2419.058493031" Sep 18 17:34:06.711824 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:06.711792 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:34:06.712930 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:06.712905 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 18 17:34:07.715864 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:07.715819 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused" Sep 18 17:34:08.721927 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:08.721890 2874 generic.go:356] "Generic (PLEG): container finished" podID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerID="5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9" exitCode=0 Sep 18 17:34:08.722396 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:08.721932 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" event={"ID":"b87d87f9-4fda-46a8-9efd-ac38b0c0d695","Type":"ContainerDied","Data":"5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9"} Sep 18 17:34:09.729085 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:09.729049 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" event={"ID":"b87d87f9-4fda-46a8-9efd-ac38b0c0d695","Type":"ContainerStarted","Data":"53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d"} Sep 18 17:34:09.729510 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:09.729113 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" event={"ID":"b87d87f9-4fda-46a8-9efd-ac38b0c0d695","Type":"ContainerStarted","Data":"e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9"} Sep 18 17:34:09.729510 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:09.729291 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:34:09.750777 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:09.750730 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podStartSLOduration=27.879110187 podStartE2EDuration="31.750713866s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.871603679s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:09.748690638 +0000 UTC m=+2423.083055386" watchObservedRunningTime="2026-09-18 17:34:09.750713866 +0000 UTC m=+2423.085078572" Sep 18 17:34:10.734921 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:10.734844 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:34:10.736573 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:10.736536 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 18 17:34:11.745751 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:11.745711 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 18 17:34:12.721199 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:12.721162 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:34:12.722184 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:12.722162 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:34:16.749848 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:16.749812 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:34:16.750461 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:16.750429 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 18 17:34:26.153611 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:26.153579 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg"] Sep 18 17:34:26.154024 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:26.153824 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kserve-container" containerID="cri-o://16f23d9a71484247e0265357951d34a6fee7d177b438659ac58b59a1ea289995" gracePeriod=30 Sep 18 17:34:26.154024 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:26.153866 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kube-rbac-proxy" containerID="cri-o://f567540db9c0166436db81b32ae8b49c57c02e1c3876de74bbd9e20adbfdda0f" gracePeriod=30 Sep 18 17:34:26.751668 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:26.751622 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 18 17:34:26.823912 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:26.823879 2874 generic.go:356] "Generic (PLEG): container finished" podID="85dbd611-d09e-4113-9bd2-530a7c919194" containerID="f567540db9c0166436db81b32ae8b49c57c02e1c3876de74bbd9e20adbfdda0f" exitCode=2 Sep 18 17:34:26.824091 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:26.823958 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" event={"ID":"85dbd611-d09e-4113-9bd2-530a7c919194","Type":"ContainerDied","Data":"f567540db9c0166436db81b32ae8b49c57c02e1c3876de74bbd9e20adbfdda0f"} Sep 18 17:34:27.716918 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:27.716879 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused" Sep 18 17:34:28.836056 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:28.835964 2874 generic.go:356] "Generic (PLEG): container finished" podID="85dbd611-d09e-4113-9bd2-530a7c919194" containerID="16f23d9a71484247e0265357951d34a6fee7d177b438659ac58b59a1ea289995" exitCode=0 Sep 18 17:34:28.836459 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:28.836046 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" event={"ID":"85dbd611-d09e-4113-9bd2-530a7c919194","Type":"ContainerDied","Data":"16f23d9a71484247e0265357951d34a6fee7d177b438659ac58b59a1ea289995"} Sep 18 17:34:29.262012 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.261980 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:34:29.271568 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.271542 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-cabundle-cert\") pod \"85dbd611-d09e-4113-9bd2-530a7c919194\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " Sep 18 17:34:29.271668 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.271584 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85dbd611-d09e-4113-9bd2-530a7c919194-proxy-tls\") pod \"85dbd611-d09e-4113-9bd2-530a7c919194\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " Sep 18 17:34:29.271668 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.271631 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85dbd611-d09e-4113-9bd2-530a7c919194-kserve-provision-location\") pod \"85dbd611-d09e-4113-9bd2-530a7c919194\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " Sep 18 17:34:29.271668 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.271661 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-kube-rbac-proxy-sar-config\") pod \"85dbd611-d09e-4113-9bd2-530a7c919194\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " Sep 18 17:34:29.271792 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.271698 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxth2\" (UniqueName: \"kubernetes.io/projected/85dbd611-d09e-4113-9bd2-530a7c919194-kube-api-access-gxth2\") pod \"85dbd611-d09e-4113-9bd2-530a7c919194\" (UID: \"85dbd611-d09e-4113-9bd2-530a7c919194\") " Sep 18 17:34:29.271890 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.271863 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85dbd611-d09e-4113-9bd2-530a7c919194" (UID: "85dbd611-d09e-4113-9bd2-530a7c919194"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:34:29.271988 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.271972 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:29.272039 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.272001 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85dbd611-d09e-4113-9bd2-530a7c919194" (UID: "85dbd611-d09e-4113-9bd2-530a7c919194"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:34:29.272080 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.272052 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dbd611-d09e-4113-9bd2-530a7c919194-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85dbd611-d09e-4113-9bd2-530a7c919194" (UID: "85dbd611-d09e-4113-9bd2-530a7c919194"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:29.274057 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.274030 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85dbd611-d09e-4113-9bd2-530a7c919194-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85dbd611-d09e-4113-9bd2-530a7c919194" (UID: "85dbd611-d09e-4113-9bd2-530a7c919194"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:34:29.274162 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.274029 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dbd611-d09e-4113-9bd2-530a7c919194-kube-api-access-gxth2" (OuterVolumeSpecName: "kube-api-access-gxth2") pod "85dbd611-d09e-4113-9bd2-530a7c919194" (UID: "85dbd611-d09e-4113-9bd2-530a7c919194"). InnerVolumeSpecName "kube-api-access-gxth2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:34:29.372853 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.372759 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85dbd611-d09e-4113-9bd2-530a7c919194-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:29.372853 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.372796 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85dbd611-d09e-4113-9bd2-530a7c919194-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:29.372853 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.372805 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxth2\" (UniqueName: \"kubernetes.io/projected/85dbd611-d09e-4113-9bd2-530a7c919194-kube-api-access-gxth2\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:29.372853 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.372815 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85dbd611-d09e-4113-9bd2-530a7c919194-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:34:29.844010 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.843984 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" Sep 18 17:34:29.844478 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.843984 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg" event={"ID":"85dbd611-d09e-4113-9bd2-530a7c919194","Type":"ContainerDied","Data":"623046f071429455ff099d946b6485055f7be0e59a93ac0a2f573e63b7d524a2"} Sep 18 17:34:29.844478 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.844123 2874 scope.go:117] "RemoveContainer" containerID="f567540db9c0166436db81b32ae8b49c57c02e1c3876de74bbd9e20adbfdda0f" Sep 18 17:34:29.853802 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.853780 2874 scope.go:117] "RemoveContainer" containerID="16f23d9a71484247e0265357951d34a6fee7d177b438659ac58b59a1ea289995" Sep 18 17:34:29.861608 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.861590 2874 scope.go:117] "RemoveContainer" containerID="eb1a47d6c9da3164eb22508c56bbc24221008ed7e3e81970c77b4bf97d18891b" Sep 18 17:34:29.867834 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.867803 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg"] Sep 18 17:34:29.873229 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:29.873208 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-md2cg"] Sep 18 17:34:30.749002 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:30.748963 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" path="/var/lib/kubelet/pods/85dbd611-d09e-4113-9bd2-530a7c919194/volumes" Sep 18 17:34:36.750805 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:36.750771 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 18 17:34:46.750367 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:46.750284 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 18 17:34:56.751192 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:34:56.751157 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 18 17:35:06.751258 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:06.751206 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 18 17:35:16.750980 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:16.750941 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 18 17:35:26.751254 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:26.751217 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused" Sep 18 17:35:36.751614 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:36.751584 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:35:38.503734 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:38.503706 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns"] Sep 18 17:35:38.504144 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:38.503910 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" containerID="cri-o://e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9" gracePeriod=30 Sep 18 17:35:38.504144 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:38.503951 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kube-rbac-proxy" containerID="cri-o://53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d" gracePeriod=30 Sep 18 17:35:39.177148 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:39.177116 2874 generic.go:356] "Generic (PLEG): container finished" podID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerID="53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d" exitCode=2 Sep 18 17:35:39.177326 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:39.177163 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" event={"ID":"b87d87f9-4fda-46a8-9efd-ac38b0c0d695","Type":"ContainerDied","Data":"53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d"} Sep 18 17:35:41.744493 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:41.744452 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused" Sep 18 17:35:43.452291 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.452258 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:35:43.487947 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.487913 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-cabundle-cert\") pod \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " Sep 18 17:35:43.488121 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.488023 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-proxy-tls\") pod \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " Sep 18 17:35:43.488121 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.488071 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-rbac-proxy-sar-config\") pod \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " Sep 18 17:35:43.488267 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.488240 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kserve-provision-location\") pod \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " Sep 18 17:35:43.488327 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.488299 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6srk9\" (UniqueName: \"kubernetes.io/projected/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-api-access-6srk9\") pod \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\" (UID: \"b87d87f9-4fda-46a8-9efd-ac38b0c0d695\") " Sep 18 17:35:43.488434 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.488389 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b87d87f9-4fda-46a8-9efd-ac38b0c0d695" (UID: "b87d87f9-4fda-46a8-9efd-ac38b0c0d695"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:35:43.488532 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.488505 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b87d87f9-4fda-46a8-9efd-ac38b0c0d695" (UID: "b87d87f9-4fda-46a8-9efd-ac38b0c0d695"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:35:43.488532 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.488513 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b87d87f9-4fda-46a8-9efd-ac38b0c0d695" (UID: "b87d87f9-4fda-46a8-9efd-ac38b0c0d695"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:35:43.488625 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.488608 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:35:43.488660 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.488626 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:35:43.488660 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.488641 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:35:43.490306 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.490285 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b87d87f9-4fda-46a8-9efd-ac38b0c0d695" (UID: "b87d87f9-4fda-46a8-9efd-ac38b0c0d695"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:35:43.490532 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.490510 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-api-access-6srk9" (OuterVolumeSpecName: "kube-api-access-6srk9") pod "b87d87f9-4fda-46a8-9efd-ac38b0c0d695" (UID: "b87d87f9-4fda-46a8-9efd-ac38b0c0d695"). InnerVolumeSpecName "kube-api-access-6srk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:35:43.589834 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.589795 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:35:43.589834 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.589827 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6srk9\" (UniqueName: \"kubernetes.io/projected/b87d87f9-4fda-46a8-9efd-ac38b0c0d695-kube-api-access-6srk9\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:35:43.792166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792078 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn"] Sep 18 17:35:43.792516 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792499 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kube-rbac-proxy" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792518 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kube-rbac-proxy" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792531 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="storage-initializer" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792540 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="storage-initializer" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792553 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="storage-initializer" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792561 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="storage-initializer" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792580 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="storage-initializer" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792588 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="storage-initializer" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792597 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792605 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792616 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kube-rbac-proxy" Sep 18 17:35:43.792628 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792624 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kube-rbac-proxy" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792636 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kube-rbac-proxy" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792644 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kube-rbac-proxy" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792656 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792665 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792685 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="storage-initializer" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792693 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="storage-initializer" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792705 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792713 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792726 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kube-rbac-proxy" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792735 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kube-rbac-proxy" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792746 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kserve-container" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792754 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kserve-container" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792845 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kserve-container" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792856 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5194fa-ac1a-4319-9a1f-3e9062796b6a" containerName="kube-rbac-proxy" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792866 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kserve-container" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792879 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="85dbd611-d09e-4113-9bd2-530a7c919194" containerName="kube-rbac-proxy" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792886 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kserve-container" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792893 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kube-rbac-proxy" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792901 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerName="kube-rbac-proxy" Sep 18 17:35:43.793166 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.792910 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="aefc9dfd-6251-41d2-a7a9-92e9ee84b149" containerName="kserve-container" Sep 18 17:35:43.795668 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.795647 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.798528 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.798508 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Sep 18 17:35:43.798613 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.798580 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\"" Sep 18 17:35:43.806030 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.806004 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn"] Sep 18 17:35:43.892160 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.892128 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0b131df-15cd-4980-9d68-42c729c3423f-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.892160 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.892164 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b131df-15cd-4980-9d68-42c729c3423f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.892384 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.892196 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.892384 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.892214 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bv5\" (UniqueName: \"kubernetes.io/projected/d0b131df-15cd-4980-9d68-42c729c3423f-kube-api-access-v9bv5\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.892384 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.892285 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.993745 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.993708 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0b131df-15cd-4980-9d68-42c729c3423f-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.993745 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.993747 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b131df-15cd-4980-9d68-42c729c3423f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.994002 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.993876 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.994002 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:35:43.993892 2874 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found Sep 18 17:35:43.994002 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.993913 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bv5\" (UniqueName: \"kubernetes.io/projected/d0b131df-15cd-4980-9d68-42c729c3423f-kube-api-access-v9bv5\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.994002 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:35:43.993974 2874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b131df-15cd-4980-9d68-42c729c3423f-proxy-tls podName:d0b131df-15cd-4980-9d68-42c729c3423f nodeName:}" failed. No retries permitted until 2026-09-18 17:35:44.493955617 +0000 UTC m=+2517.828320310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d0b131df-15cd-4980-9d68-42c729c3423f-proxy-tls") pod "isvc-canary-stop-predictor-6958d6c864-s7ztn" (UID: "d0b131df-15cd-4980-9d68-42c729c3423f") : secret "isvc-canary-stop-predictor-serving-cert" not found Sep 18 17:35:43.994194 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.994009 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.994194 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.994174 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b131df-15cd-4980-9d68-42c729c3423f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.994563 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.994547 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:43.994633 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:43.994618 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:44.005228 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.005206 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bv5\" (UniqueName: \"kubernetes.io/projected/d0b131df-15cd-4980-9d68-42c729c3423f-kube-api-access-v9bv5\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:44.204038 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.204005 2874 generic.go:356] "Generic (PLEG): container finished" podID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" containerID="e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9" exitCode=0 Sep 18 17:35:44.204218 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.204068 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" Sep 18 17:35:44.204218 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.204088 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" event={"ID":"b87d87f9-4fda-46a8-9efd-ac38b0c0d695","Type":"ContainerDied","Data":"e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9"} Sep 18 17:35:44.204218 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.204149 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns" event={"ID":"b87d87f9-4fda-46a8-9efd-ac38b0c0d695","Type":"ContainerDied","Data":"20c78b9ea650661deeba28f997f1b93e54ea77f4598b3b921e9fee92eadba7d6"} Sep 18 17:35:44.204218 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.204176 2874 scope.go:117] "RemoveContainer" containerID="53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d" Sep 18 17:35:44.213750 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.213732 2874 scope.go:117] "RemoveContainer" containerID="e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9" Sep 18 17:35:44.221057 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.221040 2874 scope.go:117] "RemoveContainer" containerID="5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9" Sep 18 17:35:44.227832 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.227810 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns"] Sep 18 17:35:44.229026 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.229012 2874 scope.go:117] "RemoveContainer" containerID="53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d" Sep 18 17:35:44.229283 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:35:44.229268 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d\": container with ID starting with 53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d not found: ID does not exist" containerID="53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d" Sep 18 17:35:44.229328 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.229291 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d"} err="failed to get container status \"53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d\": rpc error: code = NotFound desc = could not find container \"53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d\": container with ID starting with 53173f852035de781c6610ea7c75bfb818738c6ed8692a5cf6a3d8df75d8011d not found: ID does not exist" Sep 18 17:35:44.229328 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.229306 2874 scope.go:117] "RemoveContainer" containerID="e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9" Sep 18 17:35:44.229603 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:35:44.229577 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9\": container with ID starting with e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9 not found: ID does not exist" containerID="e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9" Sep 18 17:35:44.229656 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.229599 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9"} err="failed to get container status \"e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9\": rpc error: code = NotFound desc = could not find container \"e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9\": container with ID starting with e1e7b11c49dbb8efb9e530de28bab49f8e34e5586d49f409f41039b56806c4e9 not found: ID does not exist" Sep 18 17:35:44.229656 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.229614 2874 scope.go:117] "RemoveContainer" containerID="5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9" Sep 18 17:35:44.229902 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:35:44.229884 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9\": container with ID starting with 5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9 not found: ID does not exist" containerID="5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9" Sep 18 17:35:44.229983 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.229910 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9"} err="failed to get container status \"5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9\": rpc error: code = NotFound desc = could not find container \"5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9\": container with ID starting with 5d58ae76a293cc491ba5528520d3a2deed10d40358d673aac5224e3ab79cdee9 not found: ID does not exist" Sep 18 17:35:44.233309 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.233288 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-4qfns"] Sep 18 17:35:44.498152 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.498074 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0b131df-15cd-4980-9d68-42c729c3423f-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:44.500645 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.500623 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0b131df-15cd-4980-9d68-42c729c3423f-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-s7ztn\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:44.707660 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.707618 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:44.750481 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.750380 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b87d87f9-4fda-46a8-9efd-ac38b0c0d695" path="/var/lib/kubelet/pods/b87d87f9-4fda-46a8-9efd-ac38b0c0d695/volumes" Sep 18 17:35:44.983030 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:44.982986 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn"] Sep 18 17:35:44.985324 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:35:44.985295 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b131df_15cd_4980_9d68_42c729c3423f.slice/crio-98f1450ed232f3f1618b00ea9fe7db21a4518b6b189cdb8d938cf24b37528ad8 WatchSource:0}: Error finding container 98f1450ed232f3f1618b00ea9fe7db21a4518b6b189cdb8d938cf24b37528ad8: Status 404 returned error can't find the container with id 98f1450ed232f3f1618b00ea9fe7db21a4518b6b189cdb8d938cf24b37528ad8 Sep 18 17:35:45.211577 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:45.211528 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" event={"ID":"d0b131df-15cd-4980-9d68-42c729c3423f","Type":"ContainerStarted","Data":"124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656"} Sep 18 17:35:45.211765 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:45.211598 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" event={"ID":"d0b131df-15cd-4980-9d68-42c729c3423f","Type":"ContainerStarted","Data":"98f1450ed232f3f1618b00ea9fe7db21a4518b6b189cdb8d938cf24b37528ad8"} Sep 18 17:35:49.232993 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:49.232904 2874 generic.go:356] "Generic (PLEG): container finished" podID="d0b131df-15cd-4980-9d68-42c729c3423f" containerID="124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656" exitCode=0 Sep 18 17:35:49.232993 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:49.232963 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" event={"ID":"d0b131df-15cd-4980-9d68-42c729c3423f","Type":"ContainerDied","Data":"124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656"} Sep 18 17:35:50.241187 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:50.241152 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" event={"ID":"d0b131df-15cd-4980-9d68-42c729c3423f","Type":"ContainerStarted","Data":"6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8"} Sep 18 17:35:50.241591 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:50.241219 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" event={"ID":"d0b131df-15cd-4980-9d68-42c729c3423f","Type":"ContainerStarted","Data":"f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0"} Sep 18 17:35:50.241591 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:50.241419 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:50.263839 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:50.263795 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podStartSLOduration=3.346628572 podStartE2EDuration="7.263780601s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.917152029s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:35:50.262340664 +0000 UTC m=+2523.596705382" watchObservedRunningTime="2026-09-18 17:35:50.263780601 +0000 UTC m=+2523.598145308" Sep 18 17:35:51.245420 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:51.245377 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:51.247055 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:51.247028 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 18 17:35:52.249441 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:52.249386 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 18 17:35:57.254411 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:57.254365 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:35:57.255055 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:35:57.255022 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 18 17:36:07.255690 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:36:07.255651 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 18 17:36:17.255699 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:36:17.255616 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 18 17:36:27.255882 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:36:27.255835 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 18 17:36:37.255384 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:36:37.255344 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 18 17:36:47.255521 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:36:47.255481 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 18 17:36:57.255471 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:36:57.255432 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 18 17:36:59.743713 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:36:59.743672 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused" Sep 18 17:37:09.744572 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:09.744541 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:37:23.737108 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:23.737077 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn"] Sep 18 17:37:23.737495 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:23.737334 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" containerID="cri-o://f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0" gracePeriod=30 Sep 18 17:37:23.737495 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:23.737361 2874 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kube-rbac-proxy" containerID="cri-o://6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8" gracePeriod=30 Sep 18 17:37:24.676853 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:24.676821 2874 generic.go:356] "Generic (PLEG): container finished" podID="d0b131df-15cd-4980-9d68-42c729c3423f" containerID="6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8" exitCode=2 Sep 18 17:37:24.677009 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:24.676884 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" event={"ID":"d0b131df-15cd-4980-9d68-42c729c3423f","Type":"ContainerDied","Data":"6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8"} Sep 18 17:37:27.250612 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:27.250565 2874 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused" Sep 18 17:37:28.591689 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.591657 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:37:28.683335 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.683297 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b131df-15cd-4980-9d68-42c729c3423f-kserve-provision-location\") pod \"d0b131df-15cd-4980-9d68-42c729c3423f\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " Sep 18 17:37:28.683561 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.683379 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9bv5\" (UniqueName: \"kubernetes.io/projected/d0b131df-15cd-4980-9d68-42c729c3423f-kube-api-access-v9bv5\") pod \"d0b131df-15cd-4980-9d68-42c729c3423f\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " Sep 18 17:37:28.683561 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.683451 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-kube-rbac-proxy-sar-config\") pod \"d0b131df-15cd-4980-9d68-42c729c3423f\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " Sep 18 17:37:28.683561 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.683522 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0b131df-15cd-4980-9d68-42c729c3423f-proxy-tls\") pod \"d0b131df-15cd-4980-9d68-42c729c3423f\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " Sep 18 17:37:28.683845 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.683562 2874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-cabundle-cert\") pod \"d0b131df-15cd-4980-9d68-42c729c3423f\" (UID: \"d0b131df-15cd-4980-9d68-42c729c3423f\") " Sep 18 17:37:28.683845 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.683713 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b131df-15cd-4980-9d68-42c729c3423f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0b131df-15cd-4980-9d68-42c729c3423f" (UID: "d0b131df-15cd-4980-9d68-42c729c3423f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:37:28.683845 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.683821 2874 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b131df-15cd-4980-9d68-42c729c3423f-kserve-provision-location\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:37:28.684109 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.684085 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0b131df-15cd-4980-9d68-42c729c3423f" (UID: "d0b131df-15cd-4980-9d68-42c729c3423f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:37:28.684109 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.684095 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0b131df-15cd-4980-9d68-42c729c3423f" (UID: "d0b131df-15cd-4980-9d68-42c729c3423f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:37:28.685817 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.685783 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b131df-15cd-4980-9d68-42c729c3423f-kube-api-access-v9bv5" (OuterVolumeSpecName: "kube-api-access-v9bv5") pod "d0b131df-15cd-4980-9d68-42c729c3423f" (UID: "d0b131df-15cd-4980-9d68-42c729c3423f"). InnerVolumeSpecName "kube-api-access-v9bv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:37:28.685917 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.685900 2874 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0b131df-15cd-4980-9d68-42c729c3423f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0b131df-15cd-4980-9d68-42c729c3423f" (UID: "d0b131df-15cd-4980-9d68-42c729c3423f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:37:28.697866 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.697836 2874 generic.go:356] "Generic (PLEG): container finished" podID="d0b131df-15cd-4980-9d68-42c729c3423f" containerID="f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0" exitCode=0 Sep 18 17:37:28.697990 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.697878 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" event={"ID":"d0b131df-15cd-4980-9d68-42c729c3423f","Type":"ContainerDied","Data":"f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0"} Sep 18 17:37:28.697990 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.697905 2874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" Sep 18 17:37:28.697990 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.697972 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn" event={"ID":"d0b131df-15cd-4980-9d68-42c729c3423f","Type":"ContainerDied","Data":"98f1450ed232f3f1618b00ea9fe7db21a4518b6b189cdb8d938cf24b37528ad8"} Sep 18 17:37:28.698161 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.698010 2874 scope.go:117] "RemoveContainer" containerID="6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8" Sep 18 17:37:28.707390 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.707372 2874 scope.go:117] "RemoveContainer" containerID="f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0" Sep 18 17:37:28.715864 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.715847 2874 scope.go:117] "RemoveContainer" containerID="124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656" Sep 18 17:37:28.722221 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.722191 2874 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn"] Sep 18 17:37:28.725228 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.725132 2874 scope.go:117] "RemoveContainer" containerID="6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8" Sep 18 17:37:28.725632 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:37:28.725608 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8\": container with ID starting with 6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8 not found: ID does not exist" containerID="6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8" Sep 18 17:37:28.725741 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.725644 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8"} err="failed to get container status \"6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8\": rpc error: code = NotFound desc = could not find container \"6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8\": container with ID starting with 6338c7eb5e70dbabb774861e4370726374a11639fa5986d603814664d7254bb8 not found: ID does not exist" Sep 18 17:37:28.725741 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.725670 2874 scope.go:117] "RemoveContainer" containerID="f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0" Sep 18 17:37:28.726060 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:37:28.726041 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0\": container with ID starting with f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0 not found: ID does not exist" containerID="f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0" Sep 18 17:37:28.726127 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.726066 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0"} err="failed to get container status \"f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0\": rpc error: code = NotFound desc = could not find container \"f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0\": container with ID starting with f83c81c1912d4e8eed6d2b7723944fcdf5c9a89430762ca7f55a9da2537172c0 not found: ID does not exist" Sep 18 17:37:28.726127 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.726082 2874 scope.go:117] "RemoveContainer" containerID="124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656" Sep 18 17:37:28.726359 ip-10-0-132-99 kubenswrapper[2874]: E0918 17:37:28.726344 2874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656\": container with ID starting with 124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656 not found: ID does not exist" containerID="124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656" Sep 18 17:37:28.726441 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.726363 2874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656"} err="failed to get container status \"124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656\": rpc error: code = NotFound desc = could not find container \"124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656\": container with ID starting with 124ba879d74d2cc3f94ec59616e88959cdb310378daacaa4f5597cb3e7c8f656 not found: ID does not exist" Sep 18 17:37:28.727330 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.727312 2874 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-s7ztn"] Sep 18 17:37:28.749216 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.749183 2874 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" path="/var/lib/kubelet/pods/d0b131df-15cd-4980-9d68-42c729c3423f/volumes" Sep 18 17:37:28.784757 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.784726 2874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bv5\" (UniqueName: \"kubernetes.io/projected/d0b131df-15cd-4980-9d68-42c729c3423f-kube-api-access-v9bv5\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:37:28.784757 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.784749 2874 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:37:28.784757 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.784763 2874 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0b131df-15cd-4980-9d68-42c729c3423f-proxy-tls\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:37:28.784979 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:37:28.784774 2874 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b131df-15cd-4980-9d68-42c729c3423f-cabundle-cert\") on node \"ip-10-0-132-99.ec2.internal\" DevicePath \"\"" Sep 18 17:39:03.973607 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:39:03.973559 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:39:03.976145 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:39:03.976120 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:44:04.003025 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:44:04.002997 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:44:04.006504 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:44:04.006473 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:49:04.030262 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:49:04.030230 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:49:04.036563 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:49:04.036540 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:54:04.057584 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:04.057553 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:54:04.065521 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:04.065497 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:54:34.618752 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:34.618664 2874 ???:1] "http: TLS handshake error from 10.0.133.177:43948: EOF" Sep 18 17:54:34.627779 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:34.627751 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t4fzq_cb442e64-9a04-459c-814a-71a5bf49c5f7/global-pull-secret-syncer/0.log" Sep 18 17:54:34.871462 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:34.871379 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wd6bk_3c4d9103-40fb-4c3e-b830-bc31eb5edd90/konnectivity-agent/0.log" Sep 18 17:54:34.938137 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:34.938106 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-99.ec2.internal_b22dbf502d42c64e57bc1ca69e67914d/haproxy/0.log" Sep 18 17:54:39.294997 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:39.294969 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-7gbc6_a9d00046-58ab-4f9c-b9bd-85b4853f2fd8/cluster-monitoring-operator/0.log" Sep 18 17:54:39.820697 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:39.820648 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zqlqk_ae82d90e-feef-4150-a881-418cc64351dd/node-exporter/0.log" Sep 18 17:54:39.846500 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:39.846477 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zqlqk_ae82d90e-feef-4150-a881-418cc64351dd/kube-rbac-proxy/0.log" Sep 18 17:54:39.873095 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:39.873072 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zqlqk_ae82d90e-feef-4150-a881-418cc64351dd/init-textfile/0.log" Sep 18 17:54:40.382989 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:40.382959 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-jckmf_2a3737fa-5bf1-4d61-a683-7028764b81e3/prometheus-operator-admission-webhook/0.log" Sep 18 17:54:40.418161 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:40.418133 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78667b4457-5bmgn_e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1/telemeter-client/0.log" Sep 18 17:54:40.444190 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:40.444168 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78667b4457-5bmgn_e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1/reload/0.log" Sep 18 17:54:40.476076 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:40.476050 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78667b4457-5bmgn_e4e2cbff-0ae0-4f2f-9b7a-8d9f6a07c2a1/kube-rbac-proxy/0.log" Sep 18 17:54:42.077810 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.077778 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-w2jnm_1de7a1b3-6bc5-4f4c-832a-677bd92764f1/networking-console-plugin/0.log" Sep 18 17:54:42.641596 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.641564 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/1.log" Sep 18 17:54:42.649735 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.649711 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v5qkh_0d14e37c-3b2e-4192-8e48-8cc1bfba3707/console-operator/2.log" Sep 18 17:54:42.943091 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.943014 2874 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz"] Sep 18 17:54:42.943348 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.943337 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="storage-initializer" Sep 18 17:54:42.943394 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.943350 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="storage-initializer" Sep 18 17:54:42.943394 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.943363 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kube-rbac-proxy" Sep 18 17:54:42.943394 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.943369 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kube-rbac-proxy" Sep 18 17:54:42.943394 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.943381 2874 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" Sep 18 17:54:42.943394 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.943387 2874 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" Sep 18 17:54:42.943589 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.943461 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kube-rbac-proxy" Sep 18 17:54:42.943589 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.943470 2874 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0b131df-15cd-4980-9d68-42c729c3423f" containerName="kserve-container" Sep 18 17:54:42.946175 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.946161 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:42.949127 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.949105 2874 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-97c2p\"/\"default-dockercfg-h4984\"" Sep 18 17:54:42.949997 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.949978 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-97c2p\"/\"kube-root-ca.crt\"" Sep 18 17:54:42.950207 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.950194 2874 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-97c2p\"/\"openshift-service-ca.crt\"" Sep 18 17:54:42.961238 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:42.961219 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz"] Sep 18 17:54:43.109246 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.109209 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-sys\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.109666 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.109262 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvch\" (UniqueName: \"kubernetes.io/projected/caad65c1-19cc-4493-88e9-489ad895a64f-kube-api-access-6xvch\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.109666 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.109293 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-proc\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.109666 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.109318 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-lib-modules\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.109666 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.109338 2874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-podres\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.210755 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.210656 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-sys\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.210755 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.210731 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvch\" (UniqueName: \"kubernetes.io/projected/caad65c1-19cc-4493-88e9-489ad895a64f-kube-api-access-6xvch\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.210755 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.210761 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-proc\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.210992 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.210788 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-lib-modules\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.210992 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.210794 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-sys\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.210992 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.210814 2874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-podres\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.210992 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.210906 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-podres\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.210992 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.210910 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-proc\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.210992 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.210911 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caad65c1-19cc-4493-88e9-489ad895a64f-lib-modules\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.223555 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.223534 2874 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvch\" (UniqueName: \"kubernetes.io/projected/caad65c1-19cc-4493-88e9-489ad895a64f-kube-api-access-6xvch\") pod \"perf-node-gather-daemonset-rj2pz\" (UID: \"caad65c1-19cc-4493-88e9-489ad895a64f\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.256466 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.256444 2874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:43.538823 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:43.538801 2874 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz"] Sep 18 17:54:43.541103 ip-10-0-132-99 kubenswrapper[2874]: W0918 17:54:43.541077 2874 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcaad65c1_19cc_4493_88e9_489ad895a64f.slice/crio-b6d6691ebd8c5b986a05e404aa7b6279650ce29da1be2a3e7e0c36331a5698fe WatchSource:0}: Error finding container b6d6691ebd8c5b986a05e404aa7b6279650ce29da1be2a3e7e0c36331a5698fe: Status 404 returned error can't find the container with id b6d6691ebd8c5b986a05e404aa7b6279650ce29da1be2a3e7e0c36331a5698fe Sep 18 17:54:44.048547 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:44.048514 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-jzsb8_af9797ce-2519-4a42-82cc-a68378abdb3a/volume-data-source-validator/0.log" Sep 18 17:54:44.354499 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:44.354390 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" event={"ID":"caad65c1-19cc-4493-88e9-489ad895a64f","Type":"ContainerStarted","Data":"0584f437007172ef0b6a55d1d9377d9ed8b23fef7baf84cfddea316e533c7683"} Sep 18 17:54:44.354499 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:44.354468 2874 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" event={"ID":"caad65c1-19cc-4493-88e9-489ad895a64f","Type":"ContainerStarted","Data":"b6d6691ebd8c5b986a05e404aa7b6279650ce29da1be2a3e7e0c36331a5698fe"} Sep 18 17:54:44.354970 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:44.354549 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:44.374794 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:44.374754 2874 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" podStartSLOduration=2.374741208 podStartE2EDuration="2.374741208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:54:44.37367947 +0000 UTC m=+3657.708044179" watchObservedRunningTime="2026-09-18 17:54:44.374741208 +0000 UTC m=+3657.709105914" Sep 18 17:54:45.275817 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:45.275784 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gklt9_3fab1373-7e52-4716-ae69-44a8d87cc7ae/dns/0.log" Sep 18 17:54:45.302586 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:45.302550 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gklt9_3fab1373-7e52-4716-ae69-44a8d87cc7ae/kube-rbac-proxy/0.log" Sep 18 17:54:45.424873 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:45.424838 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l6cpb_175a17f1-cd97-40d2-b70a-e8ba235c28eb/dns-node-resolver/0.log" Sep 18 17:54:46.036827 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:46.036792 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-br879_eb9022da-0935-4c13-9fdc-102da03b843e/node-ca/0.log" Sep 18 17:54:46.930754 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:46.930724 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7ccfc6f4f8-ksblq_7897222e-78f1-4920-9afe-669e89dffcab/router/0.log" Sep 18 17:54:47.402297 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:47.402270 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bdv7m_8055f24c-aa2d-4eb8-adfd-7f8ce04c9bb0/serve-healthcheck-canary/0.log" Sep 18 17:54:47.826525 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:47.826415 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-phlgr_ea029ff9-8b9e-4e78-82dd-030bce98e106/insights-operator/0.log" Sep 18 17:54:47.829653 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:47.829631 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-phlgr_ea029ff9-8b9e-4e78-82dd-030bce98e106/insights-operator/1.log" Sep 18 17:54:47.851783 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:47.851762 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gg9g5_305b0ad5-11e3-4960-9666-7590e3f2e492/kube-rbac-proxy/0.log" Sep 18 17:54:47.874940 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:47.874919 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gg9g5_305b0ad5-11e3-4960-9666-7590e3f2e492/exporter/0.log" Sep 18 17:54:47.899543 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:47.899505 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gg9g5_305b0ad5-11e3-4960-9666-7590e3f2e492/extractor/0.log" Sep 18 17:54:50.369546 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:50.369517 2874 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-rj2pz" Sep 18 17:54:51.088819 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:51.088782 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5c47bdc49c-nrnp6_c27145e9-a898-4af7-96de-7d3da1bc20c7/manager/0.log" Sep 18 17:54:51.157532 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:51.157497 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-v8nnn_10b057e4-8030-4311-a307-fb2f16b3d35a/server/0.log" Sep 18 17:54:51.597256 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:51.597223 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-gpwpm_88eaa60c-05b5-4f53-8212-2e86a4b4e897/manager/0.log" Sep 18 17:54:52.022390 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:52.022368 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-pz86p_75500dae-e5dd-48cb-915b-c796da2401ba/seaweedfs/0.log" Sep 18 17:54:57.922679 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:57.922645 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-kqbn6_fea808a0-7a31-4e8c-a397-033dec7481ab/kube-storage-version-migrator-operator/1.log" Sep 18 17:54:57.925204 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:57.925175 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-kqbn6_fea808a0-7a31-4e8c-a397-033dec7481ab/kube-storage-version-migrator-operator/0.log" Sep 18 17:54:58.944686 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:58.944652 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2hbpw_16a22de8-89ae-401d-b3ff-8554161ebd3d/kube-multus/0.log" Sep 18 17:54:59.047310 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:59.047284 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bl7jm_e9465c98-ea01-4cd6-83b7-485bfb6b54f0/kube-multus-additional-cni-plugins/0.log" Sep 18 17:54:59.083127 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:59.083084 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bl7jm_e9465c98-ea01-4cd6-83b7-485bfb6b54f0/egress-router-binary-copy/0.log" Sep 18 17:54:59.121716 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:59.121690 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bl7jm_e9465c98-ea01-4cd6-83b7-485bfb6b54f0/cni-plugins/0.log" Sep 18 17:54:59.151805 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:59.151780 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bl7jm_e9465c98-ea01-4cd6-83b7-485bfb6b54f0/bond-cni-plugin/0.log" Sep 18 17:54:59.182459 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:59.182428 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bl7jm_e9465c98-ea01-4cd6-83b7-485bfb6b54f0/routeoverride-cni/0.log" Sep 18 17:54:59.207887 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:59.207792 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bl7jm_e9465c98-ea01-4cd6-83b7-485bfb6b54f0/whereabouts-cni-bincopy/0.log" Sep 18 17:54:59.234716 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:59.234690 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bl7jm_e9465c98-ea01-4cd6-83b7-485bfb6b54f0/whereabouts-cni/0.log" Sep 18 17:54:59.733962 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:59.733935 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mqrhp_bb0c72b0-0d55-4136-99ce-d50036c55d91/network-metrics-daemon/0.log" Sep 18 17:54:59.755816 ip-10-0-132-99 kubenswrapper[2874]: I0918 17:54:59.755773 2874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mqrhp_bb0c72b0-0d55-4136-99ce-d50036c55d91/kube-rbac-proxy/0.log"