Apr 16 20:35:04.892866 ip-10-0-142-90 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Apr 16 20:35:04.892878 ip-10-0-142-90 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Apr 16 20:35:04.892885 ip-10-0-142-90 systemd[1]: kubelet.service: Failed with result 'resources'. Apr 16 20:35:04.893141 ip-10-0-142-90 systemd[1]: Failed to start Kubernetes Kubelet. Apr 16 20:35:14.922256 ip-10-0-142-90 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Apr 16 20:35:14.922274 ip-10-0-142-90 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7b0c0774c21b4c69a82bfb2a82a76808 -- Apr 16 20:37:49.710878 ip-10-0-142-90 systemd[1]: Starting Kubernetes Kubelet... Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.148875 2562 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151096 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151106 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151110 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151113 2562 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151115 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151118 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151121 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 20:37:50.267089 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151123 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 20:37:50.258106 ip-10-0-142-90 systemd[1]: Started Kubernetes Kubelet. Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151126 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151129 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151131 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151134 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151136 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151139 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151141 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151144 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151146 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151149 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151152 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151157 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151161 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151163 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151166 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151168 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151171 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151173 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151176 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 20:37:50.268256 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151179 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151182 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151185 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151187 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151190 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151192 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151195 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151198 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151200 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151203 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151206 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151208 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151211 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151214 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151217 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151219 2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151221 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151224 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151229 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 20:37:50.278629 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151233 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151236 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151239 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151242 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151245 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151247 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151250 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151253 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151256 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151260 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151262 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151265 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151268 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151271 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151274 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151277 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151279 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151283 2562 feature_gate.go:328] unrecognized feature gate: Example Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151286 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 20:37:50.279938 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151288 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151291 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151293 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151296 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151298 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151301 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151303 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151305 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151308 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151311 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151317 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151320 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151323 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151325 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151328 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151331 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151333 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151336 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151338 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151341 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 20:37:50.280717 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151344 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151734 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151741 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151745 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151748 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151750 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151753 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151756 2562 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151760 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151764 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151766 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151769 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151772 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151775 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151778 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151780 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151783 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151785 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151788 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 20:37:50.282681 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151790 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151793 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151796 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151799 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151802 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151804 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151807 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151809 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151812 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151815 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151817 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151820 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151822 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151825 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151827 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151830 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151834 2562 feature_gate.go:328] unrecognized feature gate: Example Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151836 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151839 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151842 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 20:37:50.283512 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151844 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151847 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151850 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151852 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151854 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151857 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151860 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151862 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151865 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151867 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151870 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151872 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151875 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151877 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151880 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151883 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151886 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151888 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151891 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151893 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 20:37:50.284663 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151896 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151900 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151904 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151907 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151910 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151913 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151915 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151918 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151922 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151925 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151927 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151930 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151932 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151935 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151938 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151940 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151943 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151946 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151948 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 20:37:50.285443 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151951 2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151953 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151956 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151958 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151961 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151963 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151966 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151969 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.151971 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152036 2562 flags.go:64] FLAG: --address="0.0.0.0" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152044 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152049 2562 flags.go:64] FLAG: --anonymous-auth="true" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152054 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152058 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152061 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152065 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152069 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152073 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152076 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152079 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152082 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152086 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Apr 16 20:37:50.286623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152089 2562 flags.go:64] FLAG: --cgroup-root="" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152092 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152095 2562 flags.go:64] FLAG: --client-ca-file="" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152098 2562 flags.go:64] FLAG: --cloud-config="" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152101 2562 flags.go:64] FLAG: --cloud-provider="external" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152104 2562 flags.go:64] FLAG: --cluster-dns="[]" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152109 2562 flags.go:64] FLAG: --cluster-domain="" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152114 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152118 2562 flags.go:64] FLAG: --config-dir="" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152121 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152125 2562 flags.go:64] FLAG: --container-log-max-files="5" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152133 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152137 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152140 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152143 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152146 2562 flags.go:64] FLAG: --contention-profiling="false" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152149 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152152 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152155 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152158 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152162 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152166 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152169 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152171 2562 flags.go:64] FLAG: --enable-load-reader="false" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152175 2562 flags.go:64] FLAG: --enable-server="true" Apr 16 20:37:50.288508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152178 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152182 2562 flags.go:64] FLAG: --event-burst="100" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152186 2562 flags.go:64] FLAG: --event-qps="50" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152188 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152191 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152194 2562 flags.go:64] FLAG: --eviction-hard="" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152198 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152201 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152204 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152207 2562 flags.go:64] FLAG: --eviction-soft="" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152210 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152213 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152216 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152219 2562 flags.go:64] FLAG: --experimental-mounter-path="" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152223 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152226 2562 flags.go:64] FLAG: --fail-swap-on="true" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152229 2562 flags.go:64] FLAG: --feature-gates="" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152233 2562 flags.go:64] FLAG: --file-check-frequency="20s" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152236 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152240 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152243 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152246 2562 flags.go:64] FLAG: --healthz-port="10248" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152249 2562 flags.go:64] FLAG: --help="false" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152252 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152254 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Apr 16 20:37:50.289186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152257 2562 flags.go:64] FLAG: --http-check-frequency="20s" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152260 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152263 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152267 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152270 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152273 2562 flags.go:64] FLAG: --image-service-endpoint="" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152276 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152279 2562 flags.go:64] FLAG: --kube-api-burst="100" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152282 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152285 2562 flags.go:64] FLAG: --kube-api-qps="50" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152288 2562 flags.go:64] FLAG: --kube-reserved="" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152291 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152294 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152297 2562 flags.go:64] FLAG: --kubelet-cgroups="" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152299 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152302 2562 flags.go:64] FLAG: --lock-file="" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152305 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152308 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152311 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152316 2562 flags.go:64] FLAG: --log-json-split-stream="false" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152319 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152323 2562 flags.go:64] FLAG: --log-text-split-stream="false" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152326 2562 flags.go:64] FLAG: --logging-format="text" Apr 16 20:37:50.289816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152329 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152333 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152335 2562 flags.go:64] FLAG: --manifest-url="" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152338 2562 flags.go:64] FLAG: --manifest-url-header="" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152343 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152346 2562 flags.go:64] FLAG: --max-open-files="1000000" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152350 2562 flags.go:64] FLAG: --max-pods="110" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152352 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152355 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152358 2562 flags.go:64] FLAG: --memory-manager-policy="None" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152361 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152364 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152367 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152370 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152379 2562 flags.go:64] FLAG: --node-status-max-images="50" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152382 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152385 2562 flags.go:64] FLAG: --oom-score-adj="-999" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152388 2562 flags.go:64] FLAG: --pod-cidr="" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152391 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8cfe89231412ff3ee8cb6207fa0be33cad0f08e88c9c0f1e9f7e8c6f14d6715" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152397 2562 flags.go:64] FLAG: --pod-manifest-path="" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152400 2562 flags.go:64] FLAG: --pod-max-pids="-1" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152403 2562 flags.go:64] FLAG: --pods-per-core="0" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152406 2562 flags.go:64] FLAG: --port="10250" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152409 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Apr 16 20:37:50.290362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152412 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0363a06c08017c536" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152415 2562 flags.go:64] FLAG: --qos-reserved="" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152418 2562 flags.go:64] FLAG: --read-only-port="10255" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152421 2562 flags.go:64] FLAG: --register-node="true" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152424 2562 flags.go:64] FLAG: --register-schedulable="true" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152427 2562 flags.go:64] FLAG: --register-with-taints="" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152433 2562 flags.go:64] FLAG: --registry-burst="10" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152436 2562 flags.go:64] FLAG: --registry-qps="5" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152438 2562 flags.go:64] FLAG: --reserved-cpus="" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152441 2562 flags.go:64] FLAG: --reserved-memory="" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152445 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152448 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152451 2562 flags.go:64] FLAG: --rotate-certificates="false" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152454 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152456 2562 flags.go:64] FLAG: --runonce="false" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152460 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152463 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152466 2562 flags.go:64] FLAG: --seccomp-default="false" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152469 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152471 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152474 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152477 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152480 2562 flags.go:64] FLAG: --storage-driver-password="root" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152485 2562 flags.go:64] FLAG: --storage-driver-secure="false" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152488 2562 flags.go:64] FLAG: --storage-driver-table="stats" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152491 2562 flags.go:64] FLAG: --storage-driver-user="root" Apr 16 20:37:50.290946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152494 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152497 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152500 2562 flags.go:64] FLAG: --system-cgroups="" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152503 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152508 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152510 2562 flags.go:64] FLAG: --tls-cert-file="" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152513 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152520 2562 flags.go:64] FLAG: --tls-min-version="" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152523 2562 flags.go:64] FLAG: --tls-private-key-file="" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152526 2562 flags.go:64] FLAG: --topology-manager-policy="none" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152529 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152532 2562 flags.go:64] FLAG: --topology-manager-scope="container" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152536 2562 flags.go:64] FLAG: --v="2" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152540 2562 flags.go:64] FLAG: --version="false" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152544 2562 flags.go:64] FLAG: --vmodule="" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152548 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.152551 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152653 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152658 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152661 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152663 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152666 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152668 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 20:37:50.291544 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152671 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152674 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152676 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152679 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152681 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152684 2562 feature_gate.go:328] unrecognized feature gate: Example Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152688 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152696 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152699 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152701 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152704 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152706 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152709 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152712 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152714 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152717 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152719 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152722 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152725 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152727 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 20:37:50.292101 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152730 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152734 2562 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152736 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152739 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152742 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152745 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152748 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152750 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152753 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152755 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152758 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152761 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152763 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152766 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152769 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152773 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152776 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152778 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152784 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152786 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 20:37:50.292635 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152789 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152792 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152795 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152797 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152800 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152802 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152805 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152809 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152813 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152816 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152819 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152822 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152826 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152833 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152836 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152839 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152842 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152844 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152847 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 20:37:50.293117 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152849 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152852 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152855 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152857 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152860 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152863 2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152865 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152868 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152870 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152873 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152875 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152879 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152882 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152884 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152887 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152890 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152893 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152895 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152898 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152900 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 20:37:50.293583 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.152903 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.153513 2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.163691 2562 server.go:530] "Kubelet version" kubeletVersion="v1.33.9" Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.163708 2562 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163767 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163773 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163776 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163779 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163782 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163785 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163790 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163794 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163798 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163801 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163804 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 20:37:50.294077 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163814 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163817 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163820 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163822 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163825 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163827 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163830 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163832 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163835 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163838 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163840 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163843 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163846 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163848 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163851 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163853 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163856 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163859 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163861 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163864 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 20:37:50.294441 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163873 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163876 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163878 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163881 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163883 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163886 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163888 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163891 2562 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163894 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163896 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163899 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163901 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163904 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163906 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163908 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163911 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163914 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163917 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163920 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163922 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 20:37:50.294933 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163925 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163927 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163930 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163932 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163935 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163937 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163940 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163942 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163945 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163947 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163950 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163953 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163956 2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163960 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163962 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163964 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163967 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163969 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163972 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163974 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 20:37:50.295420 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163977 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163979 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163982 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163984 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163987 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163990 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163992 2562 feature_gate.go:328] unrecognized feature gate: Example Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.163997 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164001 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164004 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164007 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164010 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164013 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164015 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164018 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 20:37:50.295914 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.164023 2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164123 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164128 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164131 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164133 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164136 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164139 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164141 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164144 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164147 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164149 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164153 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164155 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164158 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164160 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164163 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164165 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164168 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164170 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164173 2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 20:37:50.296270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164175 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164178 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164181 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164184 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164187 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164190 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164192 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164195 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164198 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164200 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164202 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164205 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164208 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164210 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164213 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164215 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164218 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164220 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164224 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 20:37:50.296770 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164228 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164230 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164233 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164239 2562 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164241 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164245 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164248 2562 feature_gate.go:328] unrecognized feature gate: Example Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164250 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164252 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164255 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164258 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164260 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164262 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164266 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164269 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164272 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164275 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164279 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164281 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164284 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 20:37:50.297213 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164286 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164289 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164292 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164294 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164297 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164299 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164302 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164305 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164308 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164310 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164312 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164315 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164317 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164320 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164322 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164326 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164329 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164331 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164334 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164337 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 20:37:50.297780 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164339 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164342 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164344 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164347 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164350 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164353 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164355 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:50.164358 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.164362 2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.165067 2562 server.go:962] "Client rotation is on, will bootstrap in background" Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.166876 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.167665 2562 server.go:1019] "Starting client certificate rotation" Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.167761 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.167792 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 16 20:37:50.298294 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.192085 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 16 20:37:50.298645 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.197631 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 16 20:37:50.298645 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.215865 2562 log.go:25] "Validated CRI v1 runtime API" Apr 16 20:37:50.298645 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.221208 2562 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 16 20:37:50.298645 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.222409 2562 log.go:25] "Validated CRI v1 image API" Apr 16 20:37:50.298645 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.224530 2562 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Apr 16 20:37:50.298645 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.228817 2562 fs.go:135] Filesystem UUIDs: map[31964f52-93c8-493e-aaf6-b7f4aa97f9b1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b8cc0179-7f2b-4656-9bb1-eee7e39f9795:/dev/nvme0n1p3] Apr 16 20:37:50.298645 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.228831 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.234461 2562 manager.go:217] Machine: {Timestamp:2026-04-16 20:37:50.232315477 +0000 UTC m=+0.407679735 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098977 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2664398dfced68caa6829d7505c5e8 SystemUUID:ec266439-8dfc-ed68-caa6-829d7505c5e8 BootID:7b0c0774-c21b-4c69-a82b-fb2a82a76808 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6e:d7:08:3f:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6e:d7:08:3f:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:71:7a:e5:40:6f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.234564 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.234646 2562 manager.go:233] Version: {KernelVersion:5.14.0-570.107.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260414-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.235544 2562 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.235560 2562 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-90.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.235710 2562 topology_manager.go:138] "Creating topology manager with none policy" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.235717 2562 container_manager_linux.go:306] "Creating device plugin manager" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.235733 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.236467 2562 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.237622 2562 state_mem.go:36] "Initialized new in-memory state store" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.237728 2562 server.go:1267] "Using root directory" path="/var/lib/kubelet" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.239079 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m4jdb" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.240222 2562 kubelet.go:491] "Attempting to sync node with API server" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.240236 2562 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.240247 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.240256 2562 kubelet.go:397] "Adding apiserver pod source" Apr 16 20:37:50.298829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.240264 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.241384 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.241400 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.244023 2562 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.245172 2562 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.245958 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m4jdb" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246884 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246898 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246904 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246910 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246916 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246924 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246930 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246935 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246944 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246949 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246958 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.246967 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.248485 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.248496 2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.257313 2562 watchdog_linux.go:99] "Systemd watchdog is not enabled" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.257350 2562 server.go:1295] "Started kubelet" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.257434 2562 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.257504 2562 server_v1.go:47] "podresources" method="list" useActivePods=true Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.257448 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.258514 2562 server.go:317] "Adding debug handlers to kubelet server" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.258852 2562 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.259657 2562 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.261383 2562 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.262727 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.263462 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.264076 2562 desired_state_of_world_populator.go:150] "Desired state populator starts to run" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.264078 2562 volume_manager.go:295] "The desired_state_of_world populator starts" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.264107 2562 volume_manager.go:297] "Starting Kubelet Volume Manager" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.264170 2562 reconstruct.go:97] "Volume reconstruction finished" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.264180 2562 reconciler.go:26] "Reconciler: start to sync state" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:50.264407 2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-90.ec2.internal\" not found" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.264578 2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-90.ec2.internal" not found Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.265441 2562 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:50.267985 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-90.ec2.internal\" not found" node="ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.270512 2562 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" Apr 16 20:37:50.299276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.280298 2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-90.ec2.internal" not found Apr 16 20:37:50.337098 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.337083 2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-90.ec2.internal" not found Apr 16 20:37:50.365243 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:50.365220 2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-90.ec2.internal\" not found" Apr 16 20:37:50.389217 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.389196 2562 factory.go:55] Registering systemd factory Apr 16 20:37:50.389217 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.389223 2562 factory.go:223] Registration of the systemd container factory successfully Apr 16 20:37:50.389455 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.389441 2562 factory.go:153] Registering CRI-O factory Apr 16 20:37:50.389520 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.389458 2562 factory.go:223] Registration of the crio container factory successfully Apr 16 20:37:50.389520 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.389517 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Apr 16 20:37:50.389626 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.389548 2562 factory.go:103] Registering Raw factory Apr 16 20:37:50.389626 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.389564 2562 manager.go:1196] Started watching for new ooms in manager Apr 16 20:37:50.390142 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.390124 2562 manager.go:319] Starting recovery of all containers Apr 16 20:37:50.392807 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:50.392766 2562 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Apr 16 20:37:50.401672 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.401499 2562 manager.go:324] Recovery completed Apr 16 20:37:50.402916 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:50.402893 2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Apr 16 20:37:50.405803 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.405791 2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 20:37:50.407747 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.407732 2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-90.ec2.internal" event="NodeHasSufficientMemory" Apr 16 20:37:50.407812 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.407760 2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-90.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 20:37:50.407812 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.407770 2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-90.ec2.internal" event="NodeHasSufficientPID" Apr 16 20:37:50.408213 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.408198 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Apr 16 20:37:50.408213 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.408212 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Apr 16 20:37:50.408327 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.408228 2562 state_mem.go:36] "Initialized new in-memory state store" Apr 16 20:37:50.411414 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.411399 2562 policy_none.go:49] "None policy: Start" Apr 16 20:37:50.411414 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.411416 2562 memory_manager.go:186] "Starting memorymanager" policy="None" Apr 16 20:37:50.411509 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.411426 2562 state_mem.go:35] "Initializing new in-memory state store" Apr 16 20:37:50.452478 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.447123 2562 manager.go:341] "Starting Device Plugin manager" Apr 16 20:37:50.452478 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:50.447155 2562 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Apr 16 20:37:50.452478 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.447164 2562 server.go:85] "Starting device plugin registration server" Apr 16 20:37:50.452478 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.447366 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Apr 16 20:37:50.452478 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.447377 2562 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Apr 16 20:37:50.452478 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.447495 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Apr 16 20:37:50.452478 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.447572 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Apr 16 20:37:50.452478 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.447580 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Apr 16 20:37:50.452478 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:50.448050 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Apr 16 20:37:50.452478 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:50.448082 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-90.ec2.internal\" not found" Apr 16 20:37:50.496836 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.496815 2562 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" Apr 16 20:37:50.496917 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.496840 2562 status_manager.go:230] "Starting to sync pod status with apiserver" Apr 16 20:37:50.496917 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.496858 2562 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Apr 16 20:37:50.496917 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.496870 2562 kubelet.go:2451] "Starting kubelet main sync loop" Apr 16 20:37:50.496917 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:50.496900 2562 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Apr 16 20:37:50.500436 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.500420 2562 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 20:37:50.548409 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.548370 2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 20:37:50.549145 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.549130 2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-90.ec2.internal" event="NodeHasSufficientMemory" Apr 16 20:37:50.549247 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.549161 2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-90.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 20:37:50.549247 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.549178 2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-90.ec2.internal" event="NodeHasSufficientPID" Apr 16 20:37:50.549247 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.549213 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.555945 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.555930 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.556010 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:50.555952 2562 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-90.ec2.internal\": node \"ip-10-0-142-90.ec2.internal\" not found" Apr 16 20:37:50.597430 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.597409 2562 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal"] Apr 16 20:37:50.599728 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.599714 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.601724 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.601707 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.625186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.625166 2562 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.630977 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.630960 2562 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.639644 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.639630 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 16 20:37:50.639725 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.639629 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 16 20:37:50.667302 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.667275 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff0a205bf6c4ebd0b669be9b0baff26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal\" (UID: \"4ff0a205bf6c4ebd0b669be9b0baff26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.667302 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.667305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffbe1178dcecf35d909f25cd02be28cb-config\") pod \"kube-apiserver-proxy-ip-10-0-142-90.ec2.internal\" (UID: \"ffbe1178dcecf35d909f25cd02be28cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.667419 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.667321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ff0a205bf6c4ebd0b669be9b0baff26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal\" (UID: \"4ff0a205bf6c4ebd0b669be9b0baff26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.767427 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.767398 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ff0a205bf6c4ebd0b669be9b0baff26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal\" (UID: \"4ff0a205bf6c4ebd0b669be9b0baff26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.767427 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.767432 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff0a205bf6c4ebd0b669be9b0baff26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal\" (UID: \"4ff0a205bf6c4ebd0b669be9b0baff26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.767579 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.767451 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffbe1178dcecf35d909f25cd02be28cb-config\") pod \"kube-apiserver-proxy-ip-10-0-142-90.ec2.internal\" (UID: \"ffbe1178dcecf35d909f25cd02be28cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.767579 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.767503 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffbe1178dcecf35d909f25cd02be28cb-config\") pod \"kube-apiserver-proxy-ip-10-0-142-90.ec2.internal\" (UID: \"ffbe1178dcecf35d909f25cd02be28cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.767579 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.767504 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ff0a205bf6c4ebd0b669be9b0baff26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal\" (UID: \"4ff0a205bf6c4ebd0b669be9b0baff26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.767579 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.767536 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff0a205bf6c4ebd0b669be9b0baff26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal\" (UID: \"4ff0a205bf6c4ebd0b669be9b0baff26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.942566 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.942510 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" Apr 16 20:37:50.942566 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:50.942519 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal" Apr 16 20:37:51.167171 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.167143 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Apr 16 20:37:51.167292 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.167253 2562 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 16 20:37:51.167332 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.167295 2562 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 16 20:37:51.167332 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.167295 2562 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 16 20:37:51.241198 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.241144 2562 apiserver.go:52] "Watching apiserver" Apr 16 20:37:51.248815 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.248786 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-15 20:32:50 +0000 UTC" deadline="2027-09-10 11:19:12.956677938 +0000 UTC" Apr 16 20:37:51.248815 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.248810 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12278h41m21.707871426s" Apr 16 20:37:51.248944 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.248910 2562 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Apr 16 20:37:51.251584 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.251564 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-w8xr8","openshift-network-diagnostics/network-check-target-4d8bn","openshift-ovn-kubernetes/ovnkube-node-jx6zj","kube-system/konnectivity-agent-4jtmh","kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal","openshift-dns/node-resolver-8jxb6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal","openshift-multus/multus-additional-cni-plugins-v2b6m","openshift-multus/multus-bcnw8","openshift-network-operator/iptables-alerter-4tm6r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh","openshift-cluster-node-tuning-operator/tuned-7dm4m","openshift-image-registry/node-ca-qhhws"] Apr 16 20:37:51.256304 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.256285 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:51.256389 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.256347 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:37:51.256389 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.256375 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:51.256498 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.256441 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:37:51.258554 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.258537 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.260674 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.260653 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:37:51.261104 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.261083 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Apr 16 20:37:51.261104 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.261089 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bdfcd\"" Apr 16 20:37:51.261275 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.261134 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Apr 16 20:37:51.261275 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.261087 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Apr 16 20:37:51.262278 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.262264 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Apr 16 20:37:51.262330 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.262264 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Apr 16 20:37:51.262330 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.262317 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Apr 16 20:37:51.262766 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.262754 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.262838 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.262826 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Apr 16 20:37:51.263448 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.263427 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Apr 16 20:37:51.263539 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.263455 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7wz76\"" Apr 16 20:37:51.263539 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.263431 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Apr 16 20:37:51.264823 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.264804 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Apr 16 20:37:51.264925 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.264911 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Apr 16 20:37:51.265058 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.265042 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.267290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.265390 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nn288\"" Apr 16 20:37:51.267890 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.267875 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Apr 16 20:37:51.267975 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.267907 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Apr 16 20:37:51.267975 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.267921 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Apr 16 20:37:51.268575 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.268560 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Apr 16 20:37:51.268575 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.268569 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5jhxj\"" Apr 16 20:37:51.268700 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.268584 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Apr 16 20:37:51.269929 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.269910 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5wzs\" (UniqueName: \"kubernetes.io/projected/b8187ddd-1510-4903-bc28-7206752133c5-kube-api-access-w5wzs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:51.270012 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.269948 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-system-cni-dir\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.270012 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.269977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-os-release\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.270095 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/149ff0b8-9dee-4e72-b694-9ac834034877-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.270095 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-slash\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270095 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270058 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-etc-openvswitch\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270095 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270082 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/149ff0b8-9dee-4e72-b694-9ac834034877-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.270256 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-systemd-units\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270256 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270131 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-run-ovn\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270256 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270145 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.270256 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270151 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-log-socket\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270256 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270171 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-run-ovn-kubernetes\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270256 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270196 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270256 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270232 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfw8t\" (UniqueName: \"kubernetes.io/projected/2367260d-b94d-4df8-8908-0f55426abc92-kube-api-access-mfw8t\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270256 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270254 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24dbfe36-1150-4ee2-b906-d2d5107ac31c-tmp-dir\") pod \"node-resolver-8jxb6\" (UID: \"24dbfe36-1150-4ee2-b906-d2d5107ac31c\") " pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.270577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-cni-netd\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzjv\" (UniqueName: \"kubernetes.io/projected/24dbfe36-1150-4ee2-b906-d2d5107ac31c-kube-api-access-9tzjv\") pod \"node-resolver-8jxb6\" (UID: \"24dbfe36-1150-4ee2-b906-d2d5107ac31c\") " pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.270577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-cnibin\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.270577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270254 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.270577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270428 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2367260d-b94d-4df8-8908-0f55426abc92-ovn-node-metrics-cert\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270457 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6grv\" (UniqueName: \"kubernetes.io/projected/149ff0b8-9dee-4e72-b694-9ac834034877-kube-api-access-f6grv\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.270577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270502 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-run-netns\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270541 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-run-openvswitch\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270565 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2367260d-b94d-4df8-8908-0f55426abc92-ovnkube-config\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270625 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2367260d-b94d-4df8-8908-0f55426abc92-ovnkube-script-lib\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a66139a0-edaa-46ff-b00c-9ed2765a0ae6-agent-certs\") pod \"konnectivity-agent-4jtmh\" (UID: \"a66139a0-edaa-46ff-b00c-9ed2765a0ae6\") " pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270672 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24dbfe36-1150-4ee2-b906-d2d5107ac31c-hosts-file\") pod \"node-resolver-8jxb6\" (UID: \"24dbfe36-1150-4ee2-b906-d2d5107ac31c\") " pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npkbt\" (UniqueName: \"kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt\") pod \"network-check-target-4d8bn\" (UID: \"e5123f09-9cba-45d2-b236-0b3b4d82d9f9\") " pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-run-systemd\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-var-lib-openvswitch\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270813 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-cni-bin\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2367260d-b94d-4df8-8908-0f55426abc92-env-overrides\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270856 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/149ff0b8-9dee-4e72-b694-9ac834034877-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270880 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-kubelet\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-node-log\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.270978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.270936 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a66139a0-edaa-46ff-b00c-9ed2765a0ae6-konnectivity-ca\") pod \"konnectivity-agent-4jtmh\" (UID: \"a66139a0-edaa-46ff-b00c-9ed2765a0ae6\") " pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:37:51.272435 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.272417 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.272508 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.272442 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Apr 16 20:37:51.272564 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.272554 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kqz7s\"" Apr 16 20:37:51.272839 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.272824 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Apr 16 20:37:51.272916 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.272881 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d2jlp\"" Apr 16 20:37:51.272916 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.272886 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Apr 16 20:37:51.273545 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.273530 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Apr 16 20:37:51.274693 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.274678 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-72qsr\"" Apr 16 20:37:51.274825 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.274810 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.275166 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.275149 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Apr 16 20:37:51.275166 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.275163 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Apr 16 20:37:51.275266 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.275186 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Apr 16 20:37:51.275628 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.275597 2562 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 16 20:37:51.276847 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.276831 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.276925 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.276908 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j7x4g\"" Apr 16 20:37:51.277313 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.277296 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Apr 16 20:37:51.277398 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.277355 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Apr 16 20:37:51.278890 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.278876 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Apr 16 20:37:51.279018 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.279007 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Apr 16 20:37:51.279165 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.279152 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Apr 16 20:37:51.279309 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.279294 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-48c8z\"" Apr 16 20:37:51.291512 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.291498 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5dzdw" Apr 16 20:37:51.299138 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.299120 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5dzdw" Apr 16 20:37:51.364563 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.364545 2562 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" Apr 16 20:37:51.371275 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371257 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-cni-bin\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.371360 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371286 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2367260d-b94d-4df8-8908-0f55426abc92-env-overrides\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.371360 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371331 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-cni-bin\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.371429 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371376 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a66139a0-edaa-46ff-b00c-9ed2765a0ae6-konnectivity-ca\") pod \"konnectivity-agent-4jtmh\" (UID: \"a66139a0-edaa-46ff-b00c-9ed2765a0ae6\") " pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:37:51.371429 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371407 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-os-release\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.371514 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371430 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-run-netns\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.371514 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371458 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-node-log\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.371514 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371485 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-var-lib-kubelet\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.371660 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371548 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-node-log\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.371660 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-conf-dir\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.371731 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.371731 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.371731 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-sysconfig\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.371826 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04bd8878-b3a6-4b5d-82fa-a483f11530cc-tmp\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.371826 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371743 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2367260d-b94d-4df8-8908-0f55426abc92-env-overrides\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.371826 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8v8\" (UniqueName: \"kubernetes.io/projected/427d721c-2662-4922-a063-4b44f92e8ca2-kube-api-access-jg8v8\") pod \"node-ca-qhhws\" (UID: \"427d721c-2662-4922-a063-4b44f92e8ca2\") " pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.371826 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-cnibin\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.371974 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371833 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5wzs\" (UniqueName: \"kubernetes.io/projected/b8187ddd-1510-4903-bc28-7206752133c5-kube-api-access-w5wzs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:51.371974 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371854 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-system-cni-dir\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.371974 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371873 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-os-release\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.371974 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371897 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-slash\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.371974 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371909 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-system-cni-dir\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.371974 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371920 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-run-ovn\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.371974 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371943 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a66139a0-edaa-46ff-b00c-9ed2765a0ae6-konnectivity-ca\") pod \"konnectivity-agent-4jtmh\" (UID: \"a66139a0-edaa-46ff-b00c-9ed2765a0ae6\") " pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:37:51.371974 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371962 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-run-ovn\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.371974 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371963 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-run-ovn-kubernetes\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371996 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/149ff0b8-9dee-4e72-b694-9ac834034877-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.371995 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-slash\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372051 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-run-ovn-kubernetes\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372055 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-log-socket\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372081 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372101 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-log-socket\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372118 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24dbfe36-1150-4ee2-b906-d2d5107ac31c-tmp-dir\") pod \"node-resolver-8jxb6\" (UID: \"24dbfe36-1150-4ee2-b906-d2d5107ac31c\") " pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372126 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-os-release\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372146 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-run-k8s-cni-cncf-io\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372154 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372171 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-hostroot\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-device-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372217 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-modprobe-d\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-system-cni-dir\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-kubernetes\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-sysctl-d\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.372424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn6qw\" (UniqueName: \"kubernetes.io/projected/9466e448-57a3-4680-bc97-2db1278c6eb5-kube-api-access-hn6qw\") pod \"iptables-alerter-4tm6r\" (UID: \"9466e448-57a3-4680-bc97-2db1278c6eb5\") " pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372376 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-cnibin\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372392 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24dbfe36-1150-4ee2-b906-d2d5107ac31c-tmp-dir\") pod \"node-resolver-8jxb6\" (UID: \"24dbfe36-1150-4ee2-b906-d2d5107ac31c\") " pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372404 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2367260d-b94d-4df8-8908-0f55426abc92-ovn-node-metrics-cert\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372446 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-cnibin\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372473 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/149ff0b8-9dee-4e72-b694-9ac834034877-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-run-multus-certs\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372527 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-systemd\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-var-lib-kubelet\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372572 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/427d721c-2662-4922-a063-4b44f92e8ca2-host\") pod \"node-ca-qhhws\" (UID: \"427d721c-2662-4922-a063-4b44f92e8ca2\") " pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372590 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f6grv\" (UniqueName: \"kubernetes.io/projected/149ff0b8-9dee-4e72-b694-9ac834034877-kube-api-access-f6grv\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372623 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-socket-dir-parent\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372649 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-socket-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-sysctl-conf\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372802 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2367260d-b94d-4df8-8908-0f55426abc92-ovnkube-script-lib\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372852 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372910 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24dbfe36-1150-4ee2-b906-d2d5107ac31c-hosts-file\") pod \"node-resolver-8jxb6\" (UID: \"24dbfe36-1150-4ee2-b906-d2d5107ac31c\") " pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.373290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372859 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24dbfe36-1150-4ee2-b906-d2d5107ac31c-hosts-file\") pod \"node-resolver-8jxb6\" (UID: \"24dbfe36-1150-4ee2-b906-d2d5107ac31c\") " pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372949 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-lib-modules\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.372991 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-tuned\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373018 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npkbt\" (UniqueName: \"kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt\") pod \"network-check-target-4d8bn\" (UID: \"e5123f09-9cba-45d2-b236-0b3b4d82d9f9\") " pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373059 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373087 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-run-systemd\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373126 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-var-lib-openvswitch\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373154 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-host\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373186 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/149ff0b8-9dee-4e72-b694-9ac834034877-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373222 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-kubelet\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373266 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/149ff0b8-9dee-4e72-b694-9ac834034877-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373296 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-etc-openvswitch\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373320 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mfw8t\" (UniqueName: \"kubernetes.io/projected/2367260d-b94d-4df8-8908-0f55426abc92-kube-api-access-mfw8t\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-sys\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373371 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-systemd-units\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373393 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2367260d-b94d-4df8-8908-0f55426abc92-ovnkube-script-lib\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373399 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzjv\" (UniqueName: \"kubernetes.io/projected/24dbfe36-1150-4ee2-b906-d2d5107ac31c-kube-api-access-9tzjv\") pod \"node-resolver-8jxb6\" (UID: \"24dbfe36-1150-4ee2-b906-d2d5107ac31c\") " pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.374172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373465 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-systemd-units\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373468 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-daemon-config\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-run\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373514 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-kubelet\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373574 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-etc-openvswitch\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373594 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/149ff0b8-9dee-4e72-b694-9ac834034877-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373738 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9466e448-57a3-4680-bc97-2db1278c6eb5-host-slash\") pod \"iptables-alerter-4tm6r\" (UID: \"9466e448-57a3-4680-bc97-2db1278c6eb5\") " pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/427d721c-2662-4922-a063-4b44f92e8ca2-serviceca\") pod \"node-ca-qhhws\" (UID: \"427d721c-2662-4922-a063-4b44f92e8ca2\") " pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373798 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-var-lib-openvswitch\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373810 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-cni-netd\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373842 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-run-systemd\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373844 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/149ff0b8-9dee-4e72-b694-9ac834034877-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-cni-dir\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373890 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-cni-netd\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373918 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-cni-binary-copy\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373951 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-var-lib-cni-bin\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.373979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-etc-kubernetes\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.374794 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374017 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-registration-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374047 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-sys-fs\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qwk\" (UniqueName: \"kubernetes.io/projected/6cb537e2-bc79-4943-b35d-91d026781a9a-kube-api-access-p5qwk\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374102 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-run-netns\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9466e448-57a3-4680-bc97-2db1278c6eb5-iptables-alerter-script\") pod \"iptables-alerter-4tm6r\" (UID: \"9466e448-57a3-4680-bc97-2db1278c6eb5\") " pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374168 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-host-run-netns\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374196 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-run-openvswitch\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374215 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2367260d-b94d-4df8-8908-0f55426abc92-ovnkube-config\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374235 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/149ff0b8-9dee-4e72-b694-9ac834034877-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374248 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a66139a0-edaa-46ff-b00c-9ed2765a0ae6-agent-certs\") pod \"konnectivity-agent-4jtmh\" (UID: \"a66139a0-edaa-46ff-b00c-9ed2765a0ae6\") " pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374249 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2367260d-b94d-4df8-8908-0f55426abc92-run-openvswitch\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-var-lib-cni-multus\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374371 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksm8x\" (UniqueName: \"kubernetes.io/projected/04bd8878-b3a6-4b5d-82fa-a483f11530cc-kube-api-access-ksm8x\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5sgk\" (UniqueName: \"kubernetes.io/projected/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-kube-api-access-z5sgk\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374434 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.374559 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:37:51.375574 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.374742 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs podName:b8187ddd-1510-4903-bc28-7206752133c5 nodeName:}" failed. No retries permitted until 2026-04-16 20:37:51.874707499 +0000 UTC m=+2.050071763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs") pod "network-metrics-daemon-w8xr8" (UID: "b8187ddd-1510-4903-bc28-7206752133c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:37:51.376105 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.374761 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2367260d-b94d-4df8-8908-0f55426abc92-ovnkube-config\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.376105 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.375964 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2367260d-b94d-4df8-8908-0f55426abc92-ovn-node-metrics-cert\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.376345 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.376328 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a66139a0-edaa-46ff-b00c-9ed2765a0ae6-agent-certs\") pod \"konnectivity-agent-4jtmh\" (UID: \"a66139a0-edaa-46ff-b00c-9ed2765a0ae6\") " pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:37:51.378309 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.378287 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 20:37:51.378415 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.378314 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 20:37:51.378415 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.378327 2562 projected.go:194] Error preparing data for projected volume kube-api-access-npkbt for pod openshift-network-diagnostics/network-check-target-4d8bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:37:51.378415 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.378385 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt podName:e5123f09-9cba-45d2-b236-0b3b4d82d9f9 nodeName:}" failed. No retries permitted until 2026-04-16 20:37:51.878367457 +0000 UTC m=+2.053731720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-npkbt" (UniqueName: "kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt") pod "network-check-target-4d8bn" (UID: "e5123f09-9cba-45d2-b236-0b3b4d82d9f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:37:51.379201 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.379178 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5wzs\" (UniqueName: \"kubernetes.io/projected/b8187ddd-1510-4903-bc28-7206752133c5-kube-api-access-w5wzs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:51.379560 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.379536 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6grv\" (UniqueName: \"kubernetes.io/projected/149ff0b8-9dee-4e72-b694-9ac834034877-kube-api-access-f6grv\") pod \"multus-additional-cni-plugins-v2b6m\" (UID: \"149ff0b8-9dee-4e72-b694-9ac834034877\") " pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.380708 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.380675 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzjv\" (UniqueName: \"kubernetes.io/projected/24dbfe36-1150-4ee2-b906-d2d5107ac31c-kube-api-access-9tzjv\") pod \"node-resolver-8jxb6\" (UID: \"24dbfe36-1150-4ee2-b906-d2d5107ac31c\") " pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.381074 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.381057 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfw8t\" (UniqueName: \"kubernetes.io/projected/2367260d-b94d-4df8-8908-0f55426abc92-kube-api-access-mfw8t\") pod \"ovnkube-node-jx6zj\" (UID: \"2367260d-b94d-4df8-8908-0f55426abc92\") " pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.475333 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475313 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-var-lib-cni-multus\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475432 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475339 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ksm8x\" (UniqueName: \"kubernetes.io/projected/04bd8878-b3a6-4b5d-82fa-a483f11530cc-kube-api-access-ksm8x\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.475432 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475354 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z5sgk\" (UniqueName: \"kubernetes.io/projected/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-kube-api-access-z5sgk\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475432 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475377 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-os-release\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475432 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475422 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-var-lib-cni-multus\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475590 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475437 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-os-release\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475590 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475572 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-run-netns\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475623 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-var-lib-kubelet\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475627 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-run-netns\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475645 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-conf-dir\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475666 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.475707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475675 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-var-lib-kubelet\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475688 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475709 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-sysconfig\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475713 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-conf-dir\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475731 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04bd8878-b3a6-4b5d-82fa-a483f11530cc-tmp\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475740 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475754 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8v8\" (UniqueName: \"kubernetes.io/projected/427d721c-2662-4922-a063-4b44f92e8ca2-kube-api-access-jg8v8\") pod \"node-ca-qhhws\" (UID: \"427d721c-2662-4922-a063-4b44f92e8ca2\") " pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475760 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-sysconfig\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475767 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475776 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-cnibin\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475819 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-run-k8s-cni-cncf-io\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475832 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-cnibin\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475842 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-hostroot\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475879 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-run-k8s-cni-cncf-io\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475921 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-device-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475933 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-hostroot\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475947 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-modprobe-d\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.475989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475985 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-system-cni-dir\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476009 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-kubernetes\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476019 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-modprobe-d\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.475984 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-device-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476035 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-sysctl-d\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476062 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6qw\" (UniqueName: \"kubernetes.io/projected/9466e448-57a3-4680-bc97-2db1278c6eb5-kube-api-access-hn6qw\") pod \"iptables-alerter-4tm6r\" (UID: \"9466e448-57a3-4680-bc97-2db1278c6eb5\") " pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476065 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-kubernetes\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476093 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-run-multus-certs\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476080 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-system-cni-dir\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476118 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-systemd\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476147 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-run-multus-certs\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476150 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-sysctl-d\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476169 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-var-lib-kubelet\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476193 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/427d721c-2662-4922-a063-4b44f92e8ca2-host\") pod \"node-ca-qhhws\" (UID: \"427d721c-2662-4922-a063-4b44f92e8ca2\") " pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476202 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-systemd\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476212 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-var-lib-kubelet\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476236 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-socket-dir-parent\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476264 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/427d721c-2662-4922-a063-4b44f92e8ca2-host\") pod \"node-ca-qhhws\" (UID: \"427d721c-2662-4922-a063-4b44f92e8ca2\") " pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.476678 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476283 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-socket-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476302 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-socket-dir-parent\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476327 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-sysctl-conf\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476357 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-lib-modules\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476442 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-tuned\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476452 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-lib-modules\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476459 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-socket-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476448 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-sysctl-conf\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476492 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-host\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476519 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-sys\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476543 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-daemon-config\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476566 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-run\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476575 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-host\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476598 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-sys\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476590 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9466e448-57a3-4680-bc97-2db1278c6eb5-host-slash\") pod \"iptables-alerter-4tm6r\" (UID: \"9466e448-57a3-4680-bc97-2db1278c6eb5\") " pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476661 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04bd8878-b3a6-4b5d-82fa-a483f11530cc-run\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476668 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/427d721c-2662-4922-a063-4b44f92e8ca2-serviceca\") pod \"node-ca-qhhws\" (UID: \"427d721c-2662-4922-a063-4b44f92e8ca2\") " pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476693 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-cni-dir\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.477640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476692 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9466e448-57a3-4680-bc97-2db1278c6eb5-host-slash\") pod \"iptables-alerter-4tm6r\" (UID: \"9466e448-57a3-4680-bc97-2db1278c6eb5\") " pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476731 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-cni-binary-copy\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476742 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-cni-dir\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476758 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-var-lib-cni-bin\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476781 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-etc-kubernetes\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476804 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-registration-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476828 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-sys-fs\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476850 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qwk\" (UniqueName: \"kubernetes.io/projected/6cb537e2-bc79-4943-b35d-91d026781a9a-kube-api-access-p5qwk\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.476878 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9466e448-57a3-4680-bc97-2db1278c6eb5-iptables-alerter-script\") pod \"iptables-alerter-4tm6r\" (UID: \"9466e448-57a3-4680-bc97-2db1278c6eb5\") " pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.477056 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-multus-daemon-config\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.477128 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-registration-dir\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.477172 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-host-var-lib-cni-bin\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.477224 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-etc-kubernetes\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.477239 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cb537e2-bc79-4943-b35d-91d026781a9a-sys-fs\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.477353 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9466e448-57a3-4680-bc97-2db1278c6eb5-iptables-alerter-script\") pod \"iptables-alerter-4tm6r\" (UID: \"9466e448-57a3-4680-bc97-2db1278c6eb5\") " pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.477583 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/427d721c-2662-4922-a063-4b44f92e8ca2-serviceca\") pod \"node-ca-qhhws\" (UID: \"427d721c-2662-4922-a063-4b44f92e8ca2\") " pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.477659 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-cni-binary-copy\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.478386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.478336 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04bd8878-b3a6-4b5d-82fa-a483f11530cc-tmp\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.478961 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.478943 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04bd8878-b3a6-4b5d-82fa-a483f11530cc-etc-tuned\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.483862 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.483839 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5sgk\" (UniqueName: \"kubernetes.io/projected/005c632a-67fa-4e6a-9d1b-ab17c5f0e48d-kube-api-access-z5sgk\") pod \"multus-bcnw8\" (UID: \"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d\") " pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.484037 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.484014 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksm8x\" (UniqueName: \"kubernetes.io/projected/04bd8878-b3a6-4b5d-82fa-a483f11530cc-kube-api-access-ksm8x\") pod \"tuned-7dm4m\" (UID: \"04bd8878-b3a6-4b5d-82fa-a483f11530cc\") " pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.484299 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.484284 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qwk\" (UniqueName: \"kubernetes.io/projected/6cb537e2-bc79-4943-b35d-91d026781a9a-kube-api-access-p5qwk\") pod \"aws-ebs-csi-driver-node-9xgwh\" (UID: \"6cb537e2-bc79-4943-b35d-91d026781a9a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.485417 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.485063 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6qw\" (UniqueName: \"kubernetes.io/projected/9466e448-57a3-4680-bc97-2db1278c6eb5-kube-api-access-hn6qw\") pod \"iptables-alerter-4tm6r\" (UID: \"9466e448-57a3-4680-bc97-2db1278c6eb5\") " pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.485979 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.485873 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8v8\" (UniqueName: \"kubernetes.io/projected/427d721c-2662-4922-a063-4b44f92e8ca2-kube-api-access-jg8v8\") pod \"node-ca-qhhws\" (UID: \"427d721c-2662-4922-a063-4b44f92e8ca2\") " pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.487962 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.487941 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff0a205bf6c4ebd0b669be9b0baff26.slice/crio-cab5d64b12127ef4cc391aa6bb4a0473abb3a3482475886468d6d9df16985349 WatchSource:0}: Error finding container cab5d64b12127ef4cc391aa6bb4a0473abb3a3482475886468d6d9df16985349: Status 404 returned error can't find the container with id cab5d64b12127ef4cc391aa6bb4a0473abb3a3482475886468d6d9df16985349 Apr 16 20:37:51.488417 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.488399 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbe1178dcecf35d909f25cd02be28cb.slice/crio-47138071e35ed5bd5ea29448852d30b5d264432dedc40852a9832f04086425d6 WatchSource:0}: Error finding container 47138071e35ed5bd5ea29448852d30b5d264432dedc40852a9832f04086425d6: Status 404 returned error can't find the container with id 47138071e35ed5bd5ea29448852d30b5d264432dedc40852a9832f04086425d6 Apr 16 20:37:51.492576 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.492564 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 20:37:51.499628 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.499579 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal" event={"ID":"ffbe1178dcecf35d909f25cd02be28cb","Type":"ContainerStarted","Data":"47138071e35ed5bd5ea29448852d30b5d264432dedc40852a9832f04086425d6"} Apr 16 20:37:51.500557 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.500540 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" event={"ID":"4ff0a205bf6c4ebd0b669be9b0baff26","Type":"ContainerStarted","Data":"cab5d64b12127ef4cc391aa6bb4a0473abb3a3482475886468d6d9df16985349"} Apr 16 20:37:51.589867 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.589848 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:37:51.595135 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.595116 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2367260d_b94d_4df8_8908_0f55426abc92.slice/crio-50cf2505e37c5b29c7c8484e0f5c37ddb83da05bb349c3a3b857a22fbc2cc68e WatchSource:0}: Error finding container 50cf2505e37c5b29c7c8484e0f5c37ddb83da05bb349c3a3b857a22fbc2cc68e: Status 404 returned error can't find the container with id 50cf2505e37c5b29c7c8484e0f5c37ddb83da05bb349c3a3b857a22fbc2cc68e Apr 16 20:37:51.602801 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.602784 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:37:51.607846 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.607825 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66139a0_edaa_46ff_b00c_9ed2765a0ae6.slice/crio-f14d3dc5bfaed8ce5612c796e83812f7de119a30aa69abc4d9d767988dd5ce51 WatchSource:0}: Error finding container f14d3dc5bfaed8ce5612c796e83812f7de119a30aa69abc4d9d767988dd5ce51: Status 404 returned error can't find the container with id f14d3dc5bfaed8ce5612c796e83812f7de119a30aa69abc4d9d767988dd5ce51 Apr 16 20:37:51.622642 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.622625 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jxb6" Apr 16 20:37:51.627582 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.627563 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dbfe36_1150_4ee2_b906_d2d5107ac31c.slice/crio-62b9e1458a942f32ab403350aee9ebfb685ab5f8ca95f6fa75cab2d032cdb8ca WatchSource:0}: Error finding container 62b9e1458a942f32ab403350aee9ebfb685ab5f8ca95f6fa75cab2d032cdb8ca: Status 404 returned error can't find the container with id 62b9e1458a942f32ab403350aee9ebfb685ab5f8ca95f6fa75cab2d032cdb8ca Apr 16 20:37:51.637656 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.637640 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" Apr 16 20:37:51.643513 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.643494 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149ff0b8_9dee_4e72_b694_9ac834034877.slice/crio-baf3f5a914cae6d01f8d3a654ca89af3a60e5205e4ea949d26db2a91531df21d WatchSource:0}: Error finding container baf3f5a914cae6d01f8d3a654ca89af3a60e5205e4ea949d26db2a91531df21d: Status 404 returned error can't find the container with id baf3f5a914cae6d01f8d3a654ca89af3a60e5205e4ea949d26db2a91531df21d Apr 16 20:37:51.649250 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.649233 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bcnw8" Apr 16 20:37:51.655557 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.655536 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005c632a_67fa_4e6a_9d1b_ab17c5f0e48d.slice/crio-4608276af34703c590cbd2185642770802de8564a4b89a56473447cc6ff2860b WatchSource:0}: Error finding container 4608276af34703c590cbd2185642770802de8564a4b89a56473447cc6ff2860b: Status 404 returned error can't find the container with id 4608276af34703c590cbd2185642770802de8564a4b89a56473447cc6ff2860b Apr 16 20:37:51.667780 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.667756 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4tm6r" Apr 16 20:37:51.672921 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.672903 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9466e448_57a3_4680_bc97_2db1278c6eb5.slice/crio-63083b8dac76d82186aded65f3ad5571be1b9da604f65e5f7110ab6aaf9ff671 WatchSource:0}: Error finding container 63083b8dac76d82186aded65f3ad5571be1b9da604f65e5f7110ab6aaf9ff671: Status 404 returned error can't find the container with id 63083b8dac76d82186aded65f3ad5571be1b9da604f65e5f7110ab6aaf9ff671 Apr 16 20:37:51.673501 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.673487 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" Apr 16 20:37:51.679572 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.679554 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb537e2_bc79_4943_b35d_91d026781a9a.slice/crio-8b022bf4af7798b7b14e93496a7b7fe2d5628bd5dc2e40180337e60d49427b0e WatchSource:0}: Error finding container 8b022bf4af7798b7b14e93496a7b7fe2d5628bd5dc2e40180337e60d49427b0e: Status 404 returned error can't find the container with id 8b022bf4af7798b7b14e93496a7b7fe2d5628bd5dc2e40180337e60d49427b0e Apr 16 20:37:51.704155 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.704137 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" Apr 16 20:37:51.708848 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.708830 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bd8878_b3a6_4b5d_82fa_a483f11530cc.slice/crio-3611cfe366f4b29bed0156d4a9ed44b9b6625229ed9653dcb773e5cc936d2222 WatchSource:0}: Error finding container 3611cfe366f4b29bed0156d4a9ed44b9b6625229ed9653dcb773e5cc936d2222: Status 404 returned error can't find the container with id 3611cfe366f4b29bed0156d4a9ed44b9b6625229ed9653dcb773e5cc936d2222 Apr 16 20:37:51.708913 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.708860 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qhhws" Apr 16 20:37:51.714284 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:37:51.714264 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427d721c_2662_4922_a063_4b44f92e8ca2.slice/crio-cfef2599e7f6ba8b4ef5b5075f1350ed1275fea025c01704956b6580a2e509d3 WatchSource:0}: Error finding container cfef2599e7f6ba8b4ef5b5075f1350ed1275fea025c01704956b6580a2e509d3: Status 404 returned error can't find the container with id cfef2599e7f6ba8b4ef5b5075f1350ed1275fea025c01704956b6580a2e509d3 Apr 16 20:37:51.880232 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.880183 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:51.880232 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:51.880227 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npkbt\" (UniqueName: \"kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt\") pod \"network-check-target-4d8bn\" (UID: \"e5123f09-9cba-45d2-b236-0b3b4d82d9f9\") " pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:51.880330 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.880312 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:37:51.880368 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.880353 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 20:37:51.880368 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.880363 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs podName:b8187ddd-1510-4903-bc28-7206752133c5 nodeName:}" failed. No retries permitted until 2026-04-16 20:37:52.880348616 +0000 UTC m=+3.055712859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs") pod "network-metrics-daemon-w8xr8" (UID: "b8187ddd-1510-4903-bc28-7206752133c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:37:51.880438 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.880368 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 20:37:51.880438 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.880379 2562 projected.go:194] Error preparing data for projected volume kube-api-access-npkbt for pod openshift-network-diagnostics/network-check-target-4d8bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:37:51.880438 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:51.880412 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt podName:e5123f09-9cba-45d2-b236-0b3b4d82d9f9 nodeName:}" failed. No retries permitted until 2026-04-16 20:37:52.880401161 +0000 UTC m=+3.055765405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-npkbt" (UniqueName: "kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt") pod "network-check-target-4d8bn" (UID: "e5123f09-9cba-45d2-b236-0b3b4d82d9f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:37:52.298880 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.298810 2562 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 20:37:52.301113 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.300858 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 20:32:51 +0000 UTC" deadline="2027-10-16 09:25:23.46197146 +0000 UTC" Apr 16 20:37:52.301113 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.300883 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13140h47m31.161091976s" Apr 16 20:37:52.514683 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.514645 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qhhws" event={"ID":"427d721c-2662-4922-a063-4b44f92e8ca2","Type":"ContainerStarted","Data":"cfef2599e7f6ba8b4ef5b5075f1350ed1275fea025c01704956b6580a2e509d3"} Apr 16 20:37:52.531768 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.531694 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" event={"ID":"04bd8878-b3a6-4b5d-82fa-a483f11530cc","Type":"ContainerStarted","Data":"3611cfe366f4b29bed0156d4a9ed44b9b6625229ed9653dcb773e5cc936d2222"} Apr 16 20:37:52.545354 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.545300 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4tm6r" event={"ID":"9466e448-57a3-4680-bc97-2db1278c6eb5","Type":"ContainerStarted","Data":"63083b8dac76d82186aded65f3ad5571be1b9da604f65e5f7110ab6aaf9ff671"} Apr 16 20:37:52.567023 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.566963 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bcnw8" event={"ID":"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d","Type":"ContainerStarted","Data":"4608276af34703c590cbd2185642770802de8564a4b89a56473447cc6ff2860b"} Apr 16 20:37:52.580342 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.580319 2562 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 20:37:52.581808 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.581785 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4jtmh" event={"ID":"a66139a0-edaa-46ff-b00c-9ed2765a0ae6","Type":"ContainerStarted","Data":"f14d3dc5bfaed8ce5612c796e83812f7de119a30aa69abc4d9d767988dd5ce51"} Apr 16 20:37:52.599955 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.599935 2562 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 20:37:52.599955 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.599936 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" event={"ID":"6cb537e2-bc79-4943-b35d-91d026781a9a","Type":"ContainerStarted","Data":"8b022bf4af7798b7b14e93496a7b7fe2d5628bd5dc2e40180337e60d49427b0e"} Apr 16 20:37:52.610957 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.610933 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" event={"ID":"149ff0b8-9dee-4e72-b694-9ac834034877","Type":"ContainerStarted","Data":"baf3f5a914cae6d01f8d3a654ca89af3a60e5205e4ea949d26db2a91531df21d"} Apr 16 20:37:52.621892 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.621860 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jxb6" event={"ID":"24dbfe36-1150-4ee2-b906-d2d5107ac31c","Type":"ContainerStarted","Data":"62b9e1458a942f32ab403350aee9ebfb685ab5f8ca95f6fa75cab2d032cdb8ca"} Apr 16 20:37:52.635099 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.635056 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" event={"ID":"2367260d-b94d-4df8-8908-0f55426abc92","Type":"ContainerStarted","Data":"50cf2505e37c5b29c7c8484e0f5c37ddb83da05bb349c3a3b857a22fbc2cc68e"} Apr 16 20:37:52.889906 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.889819 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npkbt\" (UniqueName: \"kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt\") pod \"network-check-target-4d8bn\" (UID: \"e5123f09-9cba-45d2-b236-0b3b4d82d9f9\") " pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:52.889906 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:52.889883 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:52.890125 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:52.890004 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:37:52.890125 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:52.890059 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs podName:b8187ddd-1510-4903-bc28-7206752133c5 nodeName:}" failed. No retries permitted until 2026-04-16 20:37:54.890042637 +0000 UTC m=+5.065406885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs") pod "network-metrics-daemon-w8xr8" (UID: "b8187ddd-1510-4903-bc28-7206752133c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:37:52.890456 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:52.890436 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 20:37:52.890546 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:52.890461 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 20:37:52.890546 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:52.890475 2562 projected.go:194] Error preparing data for projected volume kube-api-access-npkbt for pod openshift-network-diagnostics/network-check-target-4d8bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:37:52.890546 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:52.890518 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt podName:e5123f09-9cba-45d2-b236-0b3b4d82d9f9 nodeName:}" failed. No retries permitted until 2026-04-16 20:37:54.890503942 +0000 UTC m=+5.065868195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-npkbt" (UniqueName: "kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt") pod "network-check-target-4d8bn" (UID: "e5123f09-9cba-45d2-b236-0b3b4d82d9f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:37:53.301951 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:53.301842 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 20:32:51 +0000 UTC" deadline="2027-12-11 21:25:24.531079943 +0000 UTC" Apr 16 20:37:53.301951 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:53.301877 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14496h47m31.229206635s" Apr 16 20:37:53.498010 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:53.497977 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:53.498182 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:53.498104 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:37:53.498661 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:53.498628 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:53.499668 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:53.499196 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:37:54.908570 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:54.907321 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:54.908570 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:54.907399 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npkbt\" (UniqueName: \"kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt\") pod \"network-check-target-4d8bn\" (UID: \"e5123f09-9cba-45d2-b236-0b3b4d82d9f9\") " pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:54.908570 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:54.907585 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:37:54.908570 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:54.907681 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs podName:b8187ddd-1510-4903-bc28-7206752133c5 nodeName:}" failed. No retries permitted until 2026-04-16 20:37:58.907659724 +0000 UTC m=+9.083023991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs") pod "network-metrics-daemon-w8xr8" (UID: "b8187ddd-1510-4903-bc28-7206752133c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:37:54.908570 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:54.907693 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 20:37:54.908570 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:54.907711 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 20:37:54.908570 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:54.907731 2562 projected.go:194] Error preparing data for projected volume kube-api-access-npkbt for pod openshift-network-diagnostics/network-check-target-4d8bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:37:54.908570 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:54.907789 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt podName:e5123f09-9cba-45d2-b236-0b3b4d82d9f9 nodeName:}" failed. No retries permitted until 2026-04-16 20:37:58.907764945 +0000 UTC m=+9.083129204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-npkbt" (UniqueName: "kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt") pod "network-check-target-4d8bn" (UID: "e5123f09-9cba-45d2-b236-0b3b4d82d9f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:37:55.497874 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:55.497843 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:55.498044 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:55.497955 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:37:55.498215 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:55.498192 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:55.498354 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:55.498316 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:37:57.497595 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:57.497558 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:57.498076 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:57.497630 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:57.498076 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:57.497728 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:37:57.498076 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:57.497965 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:37:58.939457 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:58.939423 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npkbt\" (UniqueName: \"kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt\") pod \"network-check-target-4d8bn\" (UID: \"e5123f09-9cba-45d2-b236-0b3b4d82d9f9\") " pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:58.939914 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:58.939485 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:58.939914 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:58.939590 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:37:58.939914 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:58.939596 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 20:37:58.939914 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:58.939637 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 20:37:58.939914 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:58.939650 2562 projected.go:194] Error preparing data for projected volume kube-api-access-npkbt for pod openshift-network-diagnostics/network-check-target-4d8bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:37:58.939914 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:58.939669 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs podName:b8187ddd-1510-4903-bc28-7206752133c5 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:06.939648601 +0000 UTC m=+17.115012860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs") pod "network-metrics-daemon-w8xr8" (UID: "b8187ddd-1510-4903-bc28-7206752133c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:37:58.939914 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:58.939700 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt podName:e5123f09-9cba-45d2-b236-0b3b4d82d9f9 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:06.939684234 +0000 UTC m=+17.115048483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-npkbt" (UniqueName: "kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt") pod "network-check-target-4d8bn" (UID: "e5123f09-9cba-45d2-b236-0b3b4d82d9f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:37:59.497244 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.497203 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:37:59.497244 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.497224 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:37:59.497479 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:59.497334 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:37:59.497537 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:59.497490 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:37:59.627755 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.627710 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mmchs"] Apr 16 20:37:59.632302 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.632272 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:37:59.632429 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:59.632357 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:37:59.745958 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.745920 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fc39fb1-0592-4764-a26a-7d30d295d17f-kubelet-config\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:37:59.746125 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.745969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fc39fb1-0592-4764-a26a-7d30d295d17f-dbus\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:37:59.746125 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.746028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:37:59.847010 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.846759 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:37:59.847010 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.846837 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fc39fb1-0592-4764-a26a-7d30d295d17f-kubelet-config\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:37:59.847010 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.846854 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fc39fb1-0592-4764-a26a-7d30d295d17f-dbus\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:37:59.847010 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.846997 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fc39fb1-0592-4764-a26a-7d30d295d17f-kubelet-config\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:37:59.847323 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:59.847011 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 20:37:59.847323 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:37:59.847032 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fc39fb1-0592-4764-a26a-7d30d295d17f-dbus\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:37:59.847323 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:37:59.847085 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret podName:3fc39fb1-0592-4764-a26a-7d30d295d17f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:00.347063367 +0000 UTC m=+10.522427622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret") pod "global-pull-secret-syncer-mmchs" (UID: "3fc39fb1-0592-4764-a26a-7d30d295d17f") : object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:00.349567 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:00.349452 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:00.350174 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:00.350144 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:00.350299 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:00.350225 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret podName:3fc39fb1-0592-4764-a26a-7d30d295d17f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:01.350206482 +0000 UTC m=+11.525570725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret") pod "global-pull-secret-syncer-mmchs" (UID: "3fc39fb1-0592-4764-a26a-7d30d295d17f") : object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:01.356549 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:01.356355 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:01.356986 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:01.356463 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:01.356986 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:01.356658 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret podName:3fc39fb1-0592-4764-a26a-7d30d295d17f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:03.356636178 +0000 UTC m=+13.532000422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret") pod "global-pull-secret-syncer-mmchs" (UID: "3fc39fb1-0592-4764-a26a-7d30d295d17f") : object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:01.497702 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:01.497671 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:01.497799 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:01.497771 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:01.497799 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:01.497785 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:01.497900 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:01.497860 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:01.497900 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:01.497868 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:01.497991 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:01.497967 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:03.371322 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:03.371285 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:03.371761 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:03.371411 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:03.371761 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:03.371473 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret podName:3fc39fb1-0592-4764-a26a-7d30d295d17f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:07.371454368 +0000 UTC m=+17.546818615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret") pod "global-pull-secret-syncer-mmchs" (UID: "3fc39fb1-0592-4764-a26a-7d30d295d17f") : object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:03.497768 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:03.497742 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:03.497918 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:03.497782 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:03.497918 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:03.497873 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:03.498036 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:03.497948 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:03.498103 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:03.498058 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:03.498157 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:03.498144 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:05.497429 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:05.497394 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:05.497862 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:05.497394 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:05.497862 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:05.497524 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:05.497862 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:05.497590 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:05.497862 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:05.497394 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:05.497862 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:05.497713 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:06.999113 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:06.999074 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npkbt\" (UniqueName: \"kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt\") pod \"network-check-target-4d8bn\" (UID: \"e5123f09-9cba-45d2-b236-0b3b4d82d9f9\") " pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:06.999543 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:06.999148 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:06.999543 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:06.999235 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 20:38:06.999543 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:06.999245 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:38:06.999543 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:06.999256 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 20:38:06.999543 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:06.999271 2562 projected.go:194] Error preparing data for projected volume kube-api-access-npkbt for pod openshift-network-diagnostics/network-check-target-4d8bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:38:06.999543 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:06.999304 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs podName:b8187ddd-1510-4903-bc28-7206752133c5 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:22.999286211 +0000 UTC m=+33.174650459 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs") pod "network-metrics-daemon-w8xr8" (UID: "b8187ddd-1510-4903-bc28-7206752133c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:38:06.999543 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:06.999323 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt podName:e5123f09-9cba-45d2-b236-0b3b4d82d9f9 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:22.999312186 +0000 UTC m=+33.174676433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-npkbt" (UniqueName: "kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt") pod "network-check-target-4d8bn" (UID: "e5123f09-9cba-45d2-b236-0b3b4d82d9f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:38:07.401762 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:07.401675 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:07.401900 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:07.401829 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:07.401900 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:07.401897 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret podName:3fc39fb1-0592-4764-a26a-7d30d295d17f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:15.401876778 +0000 UTC m=+25.577241037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret") pod "global-pull-secret-syncer-mmchs" (UID: "3fc39fb1-0592-4764-a26a-7d30d295d17f") : object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:07.497806 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:07.497770 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:07.498039 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:07.497819 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:07.498039 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:07.497891 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:07.498039 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:07.497903 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:07.498039 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:07.497995 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:07.498265 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:07.498084 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:09.497222 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:09.497164 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:09.497528 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:09.497166 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:09.497528 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:09.497255 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:09.497528 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:09.497172 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:09.497528 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:09.497332 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:09.497528 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:09.497423 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:09.660329 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:09.660296 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal" event={"ID":"ffbe1178dcecf35d909f25cd02be28cb","Type":"ContainerStarted","Data":"a7fce0870d92827d5cfb58f4522fcad9a016dfc1833ecd25488e664a39c7ebed"} Apr 16 20:38:09.661728 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:09.661701 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" event={"ID":"04bd8878-b3a6-4b5d-82fa-a483f11530cc","Type":"ContainerStarted","Data":"5670de70a7f1635380917bb37f61a88c8f2ca7c889379fc05ff396efecc305e9"} Apr 16 20:38:09.663137 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:09.663110 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bcnw8" event={"ID":"005c632a-67fa-4e6a-9d1b-ab17c5f0e48d","Type":"ContainerStarted","Data":"493eb37e81a7a14a05a4b9753f7b9570dae46c5de31a55a35cc9df2f9ced401e"} Apr 16 20:38:09.674281 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:09.674235 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-90.ec2.internal" podStartSLOduration=19.674221108 podStartE2EDuration="19.674221108s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 20:38:09.674084186 +0000 UTC m=+19.849448452" watchObservedRunningTime="2026-04-16 20:38:09.674221108 +0000 UTC m=+19.849585380" Apr 16 20:38:09.690216 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:09.690179 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7dm4m" podStartSLOduration=2.169750309 podStartE2EDuration="19.690168598s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:37:51.710255579 +0000 UTC m=+1.885619827" lastFinishedPulling="2026-04-16 20:38:09.230673868 +0000 UTC m=+19.406038116" observedRunningTime="2026-04-16 20:38:09.689793449 +0000 UTC m=+19.865157716" watchObservedRunningTime="2026-04-16 20:38:09.690168598 +0000 UTC m=+19.865532864" Apr 16 20:38:09.705200 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:09.705168 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-bcnw8" podStartSLOduration=2.133046983 podStartE2EDuration="19.705157298s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:37:51.656892476 +0000 UTC m=+1.832256720" lastFinishedPulling="2026-04-16 20:38:09.229002779 +0000 UTC m=+19.404367035" observedRunningTime="2026-04-16 20:38:09.704935903 +0000 UTC m=+19.880300168" watchObservedRunningTime="2026-04-16 20:38:09.705157298 +0000 UTC m=+19.880521563" Apr 16 20:38:10.666283 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.666127 2562 generic.go:358] "Generic (PLEG): container finished" podID="4ff0a205bf6c4ebd0b669be9b0baff26" containerID="024db6caa98ad20f20ac01f4db4dca1738c056f45c90fce68490ae573ee3c399" exitCode=0 Apr 16 20:38:10.666960 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.666216 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" event={"ID":"4ff0a205bf6c4ebd0b669be9b0baff26","Type":"ContainerDied","Data":"024db6caa98ad20f20ac01f4db4dca1738c056f45c90fce68490ae573ee3c399"} Apr 16 20:38:10.667502 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.667479 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qhhws" event={"ID":"427d721c-2662-4922-a063-4b44f92e8ca2","Type":"ContainerStarted","Data":"e1c8b38e3904bf5fee5c576166d3861ff6991a8690b76ff565049f9ed03866f4"} Apr 16 20:38:10.668747 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.668715 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4tm6r" event={"ID":"9466e448-57a3-4680-bc97-2db1278c6eb5","Type":"ContainerStarted","Data":"e9508a6bb275b8435c8b47335beb84b1c85f9cb9980213647940c033c5080bca"} Apr 16 20:38:10.670153 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.670127 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4jtmh" event={"ID":"a66139a0-edaa-46ff-b00c-9ed2765a0ae6","Type":"ContainerStarted","Data":"ddc0a41311ed31a761eac251d0587cf02d698392b5c07d7711259faabe308917"} Apr 16 20:38:10.671450 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.671424 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" event={"ID":"6cb537e2-bc79-4943-b35d-91d026781a9a","Type":"ContainerStarted","Data":"6fc4d4172ad313dcfba673993d2b113f0c5ecb0b35d09a405dfb3ee71d65f97a"} Apr 16 20:38:10.672706 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.672684 2562 generic.go:358] "Generic (PLEG): container finished" podID="149ff0b8-9dee-4e72-b694-9ac834034877" containerID="f972ecc5a0bc16c1ed138e2f460184e250b1a4490a4243fd5ac906386199d689" exitCode=0 Apr 16 20:38:10.672803 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.672739 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" event={"ID":"149ff0b8-9dee-4e72-b694-9ac834034877","Type":"ContainerDied","Data":"f972ecc5a0bc16c1ed138e2f460184e250b1a4490a4243fd5ac906386199d689"} Apr 16 20:38:10.674181 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.674155 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jxb6" event={"ID":"24dbfe36-1150-4ee2-b906-d2d5107ac31c","Type":"ContainerStarted","Data":"7e83849868c6fb53b145c3699634f224fd3a7d3d457daf89af8891272f0b0f70"} Apr 16 20:38:10.679233 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.679209 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" event={"ID":"2367260d-b94d-4df8-8908-0f55426abc92","Type":"ContainerStarted","Data":"5e37e49c1cc72e977983edf30ba633509e06a9e705b884e88c0145622551ac97"} Apr 16 20:38:10.679330 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.679243 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" event={"ID":"2367260d-b94d-4df8-8908-0f55426abc92","Type":"ContainerStarted","Data":"6880b3a87e43c0a8abafdd825014f4b27bc5fc0e5ed8da793b81ac6b8d97b1eb"} Apr 16 20:38:10.679330 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.679259 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" event={"ID":"2367260d-b94d-4df8-8908-0f55426abc92","Type":"ContainerStarted","Data":"56a7751f079189723121e4e971395830cf1ae32533d590a5b9289fff34daad7b"} Apr 16 20:38:10.679330 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.679272 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" event={"ID":"2367260d-b94d-4df8-8908-0f55426abc92","Type":"ContainerStarted","Data":"f7c85bf6f586b29ccdc38eb77e5363f0422ff1b31027df993a4f59cb77e0413b"} Apr 16 20:38:10.679330 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.679284 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" event={"ID":"2367260d-b94d-4df8-8908-0f55426abc92","Type":"ContainerStarted","Data":"6cf4f4fa77676d98ee62d5be485ebad92b307095a24ab15708c49e1ae0f4fee3"} Apr 16 20:38:10.679330 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.679296 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" event={"ID":"2367260d-b94d-4df8-8908-0f55426abc92","Type":"ContainerStarted","Data":"3da224fc85ecfedff06323fa139d1998d3e8fffbc50a4a7d11b960baabf07983"} Apr 16 20:38:10.713978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.713891 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4jtmh" podStartSLOduration=3.127775178 podStartE2EDuration="20.71387838s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:37:51.609162783 +0000 UTC m=+1.784527027" lastFinishedPulling="2026-04-16 20:38:09.19526597 +0000 UTC m=+19.370630229" observedRunningTime="2026-04-16 20:38:10.713367303 +0000 UTC m=+20.888731570" watchObservedRunningTime="2026-04-16 20:38:10.71387838 +0000 UTC m=+20.889242649" Apr 16 20:38:10.740952 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.740908 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-8jxb6" podStartSLOduration=3.174806215 podStartE2EDuration="20.740894615s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:37:51.628927434 +0000 UTC m=+1.804291678" lastFinishedPulling="2026-04-16 20:38:09.195015823 +0000 UTC m=+19.370380078" observedRunningTime="2026-04-16 20:38:10.727460704 +0000 UTC m=+20.902824971" watchObservedRunningTime="2026-04-16 20:38:10.740894615 +0000 UTC m=+20.916258883" Apr 16 20:38:10.741245 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.741215 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4tm6r" podStartSLOduration=3.220216295 podStartE2EDuration="20.741199037s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:37:51.674287264 +0000 UTC m=+1.849651508" lastFinishedPulling="2026-04-16 20:38:09.195270002 +0000 UTC m=+19.370634250" observedRunningTime="2026-04-16 20:38:10.740657262 +0000 UTC m=+20.916021532" watchObservedRunningTime="2026-04-16 20:38:10.741199037 +0000 UTC m=+20.916563305" Apr 16 20:38:10.753423 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.753231 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qhhws" podStartSLOduration=3.273621056 podStartE2EDuration="20.753216748s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:37:51.715537529 +0000 UTC m=+1.890901773" lastFinishedPulling="2026-04-16 20:38:09.195133207 +0000 UTC m=+19.370497465" observedRunningTime="2026-04-16 20:38:10.752575958 +0000 UTC m=+20.927940225" watchObservedRunningTime="2026-04-16 20:38:10.753216748 +0000 UTC m=+20.928580995" Apr 16 20:38:10.800425 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.800363 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:38:10.801006 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:10.800973 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:38:11.230972 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.230951 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Apr 16 20:38:11.455025 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.454832 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-16T20:38:11.230967748Z","UUID":"b6ef914d-3959-45d8-bd7a-6a5f5a496924","Handler":null,"Name":"","Endpoint":""} Apr 16 20:38:11.456578 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.456555 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Apr 16 20:38:11.456708 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.456584 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Apr 16 20:38:11.497394 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.497370 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:11.497489 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.497378 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:11.497489 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:11.497478 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:11.497632 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.497503 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:11.497632 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:11.497574 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:11.497731 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:11.497639 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:11.683303 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.683269 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" event={"ID":"4ff0a205bf6c4ebd0b669be9b0baff26","Type":"ContainerStarted","Data":"e2fb9694a3c727c22b5c9d4caa45a28218ea85d6c32bae1bfda53c11100bfaf6"} Apr 16 20:38:11.685223 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.685190 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" event={"ID":"6cb537e2-bc79-4943-b35d-91d026781a9a","Type":"ContainerStarted","Data":"2f2ada5e7067924cafe5898063215c3f38d9b6480335570240e55e3f10b473b3"} Apr 16 20:38:11.685720 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.685700 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:38:11.686019 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.686002 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4jtmh" Apr 16 20:38:11.712830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:11.712764 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-90.ec2.internal" podStartSLOduration=21.712752546 podStartE2EDuration="21.712752546s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 20:38:11.69878817 +0000 UTC m=+21.874152438" watchObservedRunningTime="2026-04-16 20:38:11.712752546 +0000 UTC m=+21.888116811" Apr 16 20:38:12.688276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:12.688237 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" event={"ID":"6cb537e2-bc79-4943-b35d-91d026781a9a","Type":"ContainerStarted","Data":"c7f1ab940efa2f1e1b43f34ffd01604e6486db816486d01bca9b9d7a0dbd6d08"} Apr 16 20:38:12.690899 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:12.690868 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" event={"ID":"2367260d-b94d-4df8-8908-0f55426abc92","Type":"ContainerStarted","Data":"0223fea6292aab9466ecad01aa8b6d71bf55106a19f13530f78920aa865a81ee"} Apr 16 20:38:12.704561 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:12.704525 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xgwh" podStartSLOduration=2.354483228 podStartE2EDuration="22.70451527s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:37:51.680962761 +0000 UTC m=+1.856327005" lastFinishedPulling="2026-04-16 20:38:12.03099479 +0000 UTC m=+22.206359047" observedRunningTime="2026-04-16 20:38:12.704427274 +0000 UTC m=+22.879791540" watchObservedRunningTime="2026-04-16 20:38:12.70451527 +0000 UTC m=+22.879879543" Apr 16 20:38:13.497027 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:13.496995 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:13.497027 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:13.497035 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:13.497241 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:13.497041 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:13.497241 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:13.497097 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:13.497241 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:13.497162 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:13.497241 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:13.497230 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:14.698639 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:14.698581 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" event={"ID":"2367260d-b94d-4df8-8908-0f55426abc92","Type":"ContainerStarted","Data":"d46c0b40d77aed23cda97af1e8429f7c7578b15f0f8044085d1756915317ce3b"} Apr 16 20:38:14.699519 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:14.698922 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:38:14.715100 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:14.715078 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:38:14.725353 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:14.725311 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" podStartSLOduration=6.685881674 podStartE2EDuration="24.725296584s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:37:51.596614649 +0000 UTC m=+1.771978905" lastFinishedPulling="2026-04-16 20:38:09.636029571 +0000 UTC m=+19.811393815" observedRunningTime="2026-04-16 20:38:14.72453625 +0000 UTC m=+24.899900517" watchObservedRunningTime="2026-04-16 20:38:14.725296584 +0000 UTC m=+24.900660853" Apr 16 20:38:15.473266 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:15.473167 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:15.473426 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:15.473341 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:15.473426 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:15.473408 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret podName:3fc39fb1-0592-4764-a26a-7d30d295d17f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:31.473389747 +0000 UTC m=+41.648754004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret") pod "global-pull-secret-syncer-mmchs" (UID: "3fc39fb1-0592-4764-a26a-7d30d295d17f") : object "kube-system"/"original-pull-secret" not registered Apr 16 20:38:15.502563 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:15.502073 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:15.502563 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:15.502198 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:15.502563 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:15.502344 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:15.502563 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:15.502445 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:15.502563 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:15.502474 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:15.502563 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:15.502543 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:15.700338 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:15.700311 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Apr 16 20:38:15.700798 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:15.700775 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:38:15.716417 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:15.716392 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:38:15.844946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:15.844747 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:38:16.204201 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:16.204086 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w8xr8"] Apr 16 20:38:16.204513 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:16.204490 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:16.204711 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:16.204641 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:16.204900 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:16.204842 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4d8bn"] Apr 16 20:38:16.205026 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:16.204914 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:16.205026 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:16.204997 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:16.215614 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:16.215574 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mmchs"] Apr 16 20:38:16.215703 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:16.215658 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:16.215766 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:16.215746 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:17.497418 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:17.497374 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:17.497914 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:17.497386 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:17.497914 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:17.497479 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:17.497914 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:17.497576 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:17.704312 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:17.704274 2562 generic.go:358] "Generic (PLEG): container finished" podID="149ff0b8-9dee-4e72-b694-9ac834034877" containerID="f5138cd8c196cf954eb2a2d93320c2a8acdbbfeab067e8744f3b092972c4d448" exitCode=0 Apr 16 20:38:17.704454 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:17.704374 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" event={"ID":"149ff0b8-9dee-4e72-b694-9ac834034877","Type":"ContainerDied","Data":"f5138cd8c196cf954eb2a2d93320c2a8acdbbfeab067e8744f3b092972c4d448"} Apr 16 20:38:18.500536 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:18.500506 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:18.500862 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:18.500634 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:18.708112 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:18.708081 2562 generic.go:358] "Generic (PLEG): container finished" podID="149ff0b8-9dee-4e72-b694-9ac834034877" containerID="9af8488fa70468d13c3ef8810d5b96ca70afeb6d11dcef4fd9e2fe4b98ea150c" exitCode=0 Apr 16 20:38:18.708226 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:18.708127 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" event={"ID":"149ff0b8-9dee-4e72-b694-9ac834034877","Type":"ContainerDied","Data":"9af8488fa70468d13c3ef8810d5b96ca70afeb6d11dcef4fd9e2fe4b98ea150c"} Apr 16 20:38:19.497114 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:19.497089 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:19.497204 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:19.497100 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:19.497246 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:19.497196 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:19.497296 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:19.497278 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:19.711517 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:19.711461 2562 generic.go:358] "Generic (PLEG): container finished" podID="149ff0b8-9dee-4e72-b694-9ac834034877" containerID="0875af3f2f6b0a1fd41b16d3320b6ce1dd59118d33bc199948aab79a5212ba47" exitCode=0 Apr 16 20:38:19.711517 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:19.711502 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" event={"ID":"149ff0b8-9dee-4e72-b694-9ac834034877","Type":"ContainerDied","Data":"0875af3f2f6b0a1fd41b16d3320b6ce1dd59118d33bc199948aab79a5212ba47"} Apr 16 20:38:20.500064 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:20.500029 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:20.500439 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:20.500406 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmchs" podUID="3fc39fb1-0592-4764-a26a-7d30d295d17f" Apr 16 20:38:21.498089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.498016 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:21.498089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.498034 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:21.498537 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:21.498116 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4d8bn" podUID="e5123f09-9cba-45d2-b236-0b3b4d82d9f9" Apr 16 20:38:21.498537 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:21.498258 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:38:21.662978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.662944 2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-90.ec2.internal" event="NodeReady" Apr 16 20:38:21.663126 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.663058 2562 kubelet_node_status.go:550] "Fast updating node status as it just became ready" Apr 16 20:38:21.702126 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.702096 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vxpfw"] Apr 16 20:38:21.705265 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.705234 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.706338 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.706316 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qtfdj"] Apr 16 20:38:21.707998 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.707975 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wwh4f\"" Apr 16 20:38:21.708085 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.707990 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Apr 16 20:38:21.708259 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.708242 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Apr 16 20:38:21.709240 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.709223 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:21.711709 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.711682 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Apr 16 20:38:21.711709 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.711704 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Apr 16 20:38:21.711848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.711752 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7vq8v\"" Apr 16 20:38:21.711848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.711688 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Apr 16 20:38:21.718054 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.718004 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vxpfw"] Apr 16 20:38:21.718812 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.718792 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qtfdj"] Apr 16 20:38:21.825222 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.825192 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c72f51ec-57bd-429f-85d6-aa06c2c66728-tmp-dir\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.825350 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.825230 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f51ec-57bd-429f-85d6-aa06c2c66728-config-volume\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.825350 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.825249 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:21.825350 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.825306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44qgn\" (UniqueName: \"kubernetes.io/projected/6781237f-45c1-4874-a81d-43042624470f-kube-api-access-44qgn\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:21.825350 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.825337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.825507 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.825362 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzfr\" (UniqueName: \"kubernetes.io/projected/c72f51ec-57bd-429f-85d6-aa06c2c66728-kube-api-access-2zzfr\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.926016 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.925990 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c72f51ec-57bd-429f-85d6-aa06c2c66728-tmp-dir\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.926016 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.926018 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f51ec-57bd-429f-85d6-aa06c2c66728-config-volume\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.926201 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.926036 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:21.926201 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.926075 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44qgn\" (UniqueName: \"kubernetes.io/projected/6781237f-45c1-4874-a81d-43042624470f-kube-api-access-44qgn\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:21.926201 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.926111 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.926201 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.926138 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzfr\" (UniqueName: \"kubernetes.io/projected/c72f51ec-57bd-429f-85d6-aa06c2c66728-kube-api-access-2zzfr\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.926201 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:21.926181 2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 20:38:21.926437 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:21.926217 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 20:38:21.926437 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:21.926247 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert podName:6781237f-45c1-4874-a81d-43042624470f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:22.426228628 +0000 UTC m=+32.601592874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert") pod "ingress-canary-qtfdj" (UID: "6781237f-45c1-4874-a81d-43042624470f") : secret "canary-serving-cert" not found Apr 16 20:38:21.926437 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:21.926273 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls podName:c72f51ec-57bd-429f-85d6-aa06c2c66728 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:22.426255472 +0000 UTC m=+32.601619721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls") pod "dns-default-vxpfw" (UID: "c72f51ec-57bd-429f-85d6-aa06c2c66728") : secret "dns-default-metrics-tls" not found Apr 16 20:38:21.926552 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.926489 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c72f51ec-57bd-429f-85d6-aa06c2c66728-tmp-dir\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.926818 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.926798 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f51ec-57bd-429f-85d6-aa06c2c66728-config-volume\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.938837 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.938819 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzfr\" (UniqueName: \"kubernetes.io/projected/c72f51ec-57bd-429f-85d6-aa06c2c66728-kube-api-access-2zzfr\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:21.939569 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:21.939538 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44qgn\" (UniqueName: \"kubernetes.io/projected/6781237f-45c1-4874-a81d-43042624470f-kube-api-access-44qgn\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:22.429329 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:22.429267 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:22.429514 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:22.429362 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:22.429514 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:22.429428 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 20:38:22.429514 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:22.429474 2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 20:38:22.429514 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:22.429506 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls podName:c72f51ec-57bd-429f-85d6-aa06c2c66728 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:23.429487103 +0000 UTC m=+33.604851354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls") pod "dns-default-vxpfw" (UID: "c72f51ec-57bd-429f-85d6-aa06c2c66728") : secret "dns-default-metrics-tls" not found Apr 16 20:38:22.429711 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:22.429524 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert podName:6781237f-45c1-4874-a81d-43042624470f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:23.429515498 +0000 UTC m=+33.604879742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert") pod "ingress-canary-qtfdj" (UID: "6781237f-45c1-4874-a81d-43042624470f") : secret "canary-serving-cert" not found Apr 16 20:38:22.497351 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:22.497321 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:22.500539 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:22.500454 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Apr 16 20:38:23.035077 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.035031 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:23.035258 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:23.035221 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:38:23.035258 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.035230 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npkbt\" (UniqueName: \"kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt\") pod \"network-check-target-4d8bn\" (UID: \"e5123f09-9cba-45d2-b236-0b3b4d82d9f9\") " pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:23.035376 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:23.035296 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs podName:b8187ddd-1510-4903-bc28-7206752133c5 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:55.035276004 +0000 UTC m=+65.210640250 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs") pod "network-metrics-daemon-w8xr8" (UID: "b8187ddd-1510-4903-bc28-7206752133c5") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 20:38:23.035376 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:23.035372 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 20:38:23.035488 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:23.035388 2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 20:38:23.035488 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:23.035401 2562 projected.go:194] Error preparing data for projected volume kube-api-access-npkbt for pod openshift-network-diagnostics/network-check-target-4d8bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:38:23.035488 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:23.035456 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt podName:e5123f09-9cba-45d2-b236-0b3b4d82d9f9 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:55.035441058 +0000 UTC m=+65.210805306 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-npkbt" (UniqueName: "kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt") pod "network-check-target-4d8bn" (UID: "e5123f09-9cba-45d2-b236-0b3b4d82d9f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 20:38:23.437516 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.437481 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:23.437707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.437549 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:23.437707 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:23.437662 2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 20:38:23.437829 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:23.437738 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert podName:6781237f-45c1-4874-a81d-43042624470f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:25.43771579 +0000 UTC m=+35.613080038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert") pod "ingress-canary-qtfdj" (UID: "6781237f-45c1-4874-a81d-43042624470f") : secret "canary-serving-cert" not found Apr 16 20:38:23.437829 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:23.437667 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 20:38:23.437918 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:23.437844 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls podName:c72f51ec-57bd-429f-85d6-aa06c2c66728 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:25.437823389 +0000 UTC m=+35.613187641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls") pod "dns-default-vxpfw" (UID: "c72f51ec-57bd-429f-85d6-aa06c2c66728") : secret "dns-default-metrics-tls" not found Apr 16 20:38:23.497421 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.497391 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:23.497573 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.497391 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:23.500466 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.500441 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 16 20:38:23.502058 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.501889 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 16 20:38:23.502058 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.501896 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-484zf\"" Apr 16 20:38:23.502058 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.501945 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vdj2n\"" Apr 16 20:38:23.502058 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:23.501989 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 16 20:38:25.454801 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:25.454775 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:25.455263 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:25.454825 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:25.455263 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:25.454915 2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 20:38:25.455263 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:25.454940 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 20:38:25.455263 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:25.454969 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert podName:6781237f-45c1-4874-a81d-43042624470f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:29.45495534 +0000 UTC m=+39.630319587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert") pod "ingress-canary-qtfdj" (UID: "6781237f-45c1-4874-a81d-43042624470f") : secret "canary-serving-cert" not found Apr 16 20:38:25.455263 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:25.454983 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls podName:c72f51ec-57bd-429f-85d6-aa06c2c66728 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:29.454976826 +0000 UTC m=+39.630341070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls") pod "dns-default-vxpfw" (UID: "c72f51ec-57bd-429f-85d6-aa06c2c66728") : secret "dns-default-metrics-tls" not found Apr 16 20:38:26.725119 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:26.725086 2562 generic.go:358] "Generic (PLEG): container finished" podID="149ff0b8-9dee-4e72-b694-9ac834034877" containerID="a487ed46ad357054a6c0b7424121bdc5af79940363b521718a4df0e04c2ce020" exitCode=0 Apr 16 20:38:26.725534 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:26.725142 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" event={"ID":"149ff0b8-9dee-4e72-b694-9ac834034877","Type":"ContainerDied","Data":"a487ed46ad357054a6c0b7424121bdc5af79940363b521718a4df0e04c2ce020"} Apr 16 20:38:27.730017 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:27.729977 2562 generic.go:358] "Generic (PLEG): container finished" podID="149ff0b8-9dee-4e72-b694-9ac834034877" containerID="d1b192edd66de22f8ce6efd88ed9a56e3cd5274d0e6e8ac1c6c2960a8265e03f" exitCode=0 Apr 16 20:38:27.730437 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:27.730047 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" event={"ID":"149ff0b8-9dee-4e72-b694-9ac834034877","Type":"ContainerDied","Data":"d1b192edd66de22f8ce6efd88ed9a56e3cd5274d0e6e8ac1c6c2960a8265e03f"} Apr 16 20:38:28.735167 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:28.735134 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" event={"ID":"149ff0b8-9dee-4e72-b694-9ac834034877","Type":"ContainerStarted","Data":"c0c665d720652c66f3e174d46c12088195c935a4bcc22047fe509a4c6e355778"} Apr 16 20:38:28.757839 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:28.757797 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v2b6m" podStartSLOduration=4.7396829369999995 podStartE2EDuration="38.757784544s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:37:51.644893902 +0000 UTC m=+1.820258149" lastFinishedPulling="2026-04-16 20:38:25.66299551 +0000 UTC m=+35.838359756" observedRunningTime="2026-04-16 20:38:28.756239623 +0000 UTC m=+38.931603890" watchObservedRunningTime="2026-04-16 20:38:28.757784544 +0000 UTC m=+38.933148810" Apr 16 20:38:29.487955 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:29.487923 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:29.488121 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:29.487981 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:29.488121 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:29.488081 2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 20:38:29.488121 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:29.488092 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 20:38:29.488220 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:29.488145 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert podName:6781237f-45c1-4874-a81d-43042624470f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:37.48813073 +0000 UTC m=+47.663494977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert") pod "ingress-canary-qtfdj" (UID: "6781237f-45c1-4874-a81d-43042624470f") : secret "canary-serving-cert" not found Apr 16 20:38:29.488220 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:29.488167 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls podName:c72f51ec-57bd-429f-85d6-aa06c2c66728 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:37.488161547 +0000 UTC m=+47.663525791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls") pod "dns-default-vxpfw" (UID: "c72f51ec-57bd-429f-85d6-aa06c2c66728") : secret "dns-default-metrics-tls" not found Apr 16 20:38:31.498638 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:31.498584 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:31.502187 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:31.502161 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fc39fb1-0592-4764-a26a-7d30d295d17f-original-pull-secret\") pod \"global-pull-secret-syncer-mmchs\" (UID: \"3fc39fb1-0592-4764-a26a-7d30d295d17f\") " pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:31.508080 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:31.508055 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmchs" Apr 16 20:38:31.677161 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:31.677132 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mmchs"] Apr 16 20:38:31.681160 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:38:31.681129 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc39fb1_0592_4764_a26a_7d30d295d17f.slice/crio-268cbed14115365d9c64e0c608fb9de63bbf3c3780cd82de16fed9fb5257954f WatchSource:0}: Error finding container 268cbed14115365d9c64e0c608fb9de63bbf3c3780cd82de16fed9fb5257954f: Status 404 returned error can't find the container with id 268cbed14115365d9c64e0c608fb9de63bbf3c3780cd82de16fed9fb5257954f Apr 16 20:38:31.742469 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:31.742441 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mmchs" event={"ID":"3fc39fb1-0592-4764-a26a-7d30d295d17f","Type":"ContainerStarted","Data":"268cbed14115365d9c64e0c608fb9de63bbf3c3780cd82de16fed9fb5257954f"} Apr 16 20:38:36.752493 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:36.752452 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mmchs" event={"ID":"3fc39fb1-0592-4764-a26a-7d30d295d17f","Type":"ContainerStarted","Data":"87220dcbbf275e7130ebbacb04281ea1828ac3c5e4f787adbfaf9d78609910c1"} Apr 16 20:38:36.767243 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:36.767196 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mmchs" podStartSLOduration=33.538289116 podStartE2EDuration="37.767184619s" podCreationTimestamp="2026-04-16 20:37:59 +0000 UTC" firstStartedPulling="2026-04-16 20:38:31.682626674 +0000 UTC m=+41.857990918" lastFinishedPulling="2026-04-16 20:38:35.911522175 +0000 UTC m=+46.086886421" observedRunningTime="2026-04-16 20:38:36.767010266 +0000 UTC m=+46.942374533" watchObservedRunningTime="2026-04-16 20:38:36.767184619 +0000 UTC m=+46.942548885" Apr 16 20:38:37.538122 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:37.538094 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:37.538264 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:37.538140 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:37.538264 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:37.538230 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 20:38:37.538336 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:37.538274 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls podName:c72f51ec-57bd-429f-85d6-aa06c2c66728 nodeName:}" failed. No retries permitted until 2026-04-16 20:38:53.538261148 +0000 UTC m=+63.713625391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls") pod "dns-default-vxpfw" (UID: "c72f51ec-57bd-429f-85d6-aa06c2c66728") : secret "dns-default-metrics-tls" not found Apr 16 20:38:37.538336 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:37.538230 2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 20:38:37.538402 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:37.538352 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert podName:6781237f-45c1-4874-a81d-43042624470f nodeName:}" failed. No retries permitted until 2026-04-16 20:38:53.538339925 +0000 UTC m=+63.713704174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert") pod "ingress-canary-qtfdj" (UID: "6781237f-45c1-4874-a81d-43042624470f") : secret "canary-serving-cert" not found Apr 16 20:38:47.719100 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:47.719071 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jx6zj" Apr 16 20:38:53.538623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:53.538572 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:38:53.539034 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:53.538648 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:38:53.539034 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:53.538731 2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 20:38:53.539034 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:53.538769 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 20:38:53.539034 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:53.538804 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert podName:6781237f-45c1-4874-a81d-43042624470f nodeName:}" failed. No retries permitted until 2026-04-16 20:39:25.538787 +0000 UTC m=+95.714151244 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert") pod "ingress-canary-qtfdj" (UID: "6781237f-45c1-4874-a81d-43042624470f") : secret "canary-serving-cert" not found Apr 16 20:38:53.539034 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:53.538830 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls podName:c72f51ec-57bd-429f-85d6-aa06c2c66728 nodeName:}" failed. No retries permitted until 2026-04-16 20:39:25.538812092 +0000 UTC m=+95.714176336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls") pod "dns-default-vxpfw" (UID: "c72f51ec-57bd-429f-85d6-aa06c2c66728") : secret "dns-default-metrics-tls" not found Apr 16 20:38:55.049324 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:55.049284 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npkbt\" (UniqueName: \"kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt\") pod \"network-check-target-4d8bn\" (UID: \"e5123f09-9cba-45d2-b236-0b3b4d82d9f9\") " pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:55.049690 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:55.049348 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:38:55.051990 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:55.051973 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 16 20:38:55.052175 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:55.052163 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 16 20:38:55.059519 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:55.059504 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 16 20:38:55.059578 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:38:55.059552 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs podName:b8187ddd-1510-4903-bc28-7206752133c5 nodeName:}" failed. No retries permitted until 2026-04-16 20:39:59.059539455 +0000 UTC m=+129.234903699 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs") pod "network-metrics-daemon-w8xr8" (UID: "b8187ddd-1510-4903-bc28-7206752133c5") : secret "metrics-daemon-secret" not found Apr 16 20:38:55.061780 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:55.061765 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 16 20:38:55.072234 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:55.072218 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npkbt\" (UniqueName: \"kubernetes.io/projected/e5123f09-9cba-45d2-b236-0b3b4d82d9f9-kube-api-access-npkbt\") pod \"network-check-target-4d8bn\" (UID: \"e5123f09-9cba-45d2-b236-0b3b4d82d9f9\") " pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:55.311572 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:55.311523 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-484zf\"" Apr 16 20:38:55.319675 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:55.319660 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:55.426664 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:55.426630 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4d8bn"] Apr 16 20:38:55.430509 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:38:55.430479 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5123f09_9cba_45d2_b236_0b3b4d82d9f9.slice/crio-821f2e6d5f62cf46396a17d9330f5694c5432ec9923c3a4ea6ec4dee8189791a WatchSource:0}: Error finding container 821f2e6d5f62cf46396a17d9330f5694c5432ec9923c3a4ea6ec4dee8189791a: Status 404 returned error can't find the container with id 821f2e6d5f62cf46396a17d9330f5694c5432ec9923c3a4ea6ec4dee8189791a Apr 16 20:38:55.785582 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:55.785522 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4d8bn" event={"ID":"e5123f09-9cba-45d2-b236-0b3b4d82d9f9","Type":"ContainerStarted","Data":"821f2e6d5f62cf46396a17d9330f5694c5432ec9923c3a4ea6ec4dee8189791a"} Apr 16 20:38:59.794292 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:59.794257 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4d8bn" event={"ID":"e5123f09-9cba-45d2-b236-0b3b4d82d9f9","Type":"ContainerStarted","Data":"9b5239779056994e04771087f7ce174620cb2bbd62b92a50d95636986c24ee6c"} Apr 16 20:38:59.794677 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:59.794381 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:38:59.810844 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:38:59.810806 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4d8bn" podStartSLOduration=66.31357997 podStartE2EDuration="1m9.810793491s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:38:55.432345483 +0000 UTC m=+65.607709727" lastFinishedPulling="2026-04-16 20:38:58.929558999 +0000 UTC m=+69.104923248" observedRunningTime="2026-04-16 20:38:59.809860247 +0000 UTC m=+69.985224512" watchObservedRunningTime="2026-04-16 20:38:59.810793491 +0000 UTC m=+69.986157757" Apr 16 20:39:25.545691 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:25.545645 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:39:25.545691 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:25.545704 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:39:25.546311 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:25.545811 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 20:39:25.546311 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:25.545906 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls podName:c72f51ec-57bd-429f-85d6-aa06c2c66728 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:29.545887425 +0000 UTC m=+159.721251669 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls") pod "dns-default-vxpfw" (UID: "c72f51ec-57bd-429f-85d6-aa06c2c66728") : secret "dns-default-metrics-tls" not found Apr 16 20:39:25.546311 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:25.545811 2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 20:39:25.546311 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:25.545957 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert podName:6781237f-45c1-4874-a81d-43042624470f nodeName:}" failed. No retries permitted until 2026-04-16 20:40:29.545944292 +0000 UTC m=+159.721308551 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert") pod "ingress-canary-qtfdj" (UID: "6781237f-45c1-4874-a81d-43042624470f") : secret "canary-serving-cert" not found Apr 16 20:39:30.799507 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:30.799477 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4d8bn" Apr 16 20:39:56.855766 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.855732 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf"] Apr 16 20:39:56.858615 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.858585 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:39:56.861278 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.861241 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Apr 16 20:39:56.861400 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.861243 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kw4h2\"" Apr 16 20:39:56.862432 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.862409 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Apr 16 20:39:56.862528 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.862437 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Apr 16 20:39:56.868053 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.868033 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf"] Apr 16 20:39:56.938093 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.938068 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjb4\" (UniqueName: \"kubernetes.io/projected/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-kube-api-access-vwjb4\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:39:56.938191 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.938142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:39:56.968088 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.968053 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-9d4b6777b-s4bp5"] Apr 16 20:39:56.970683 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.970668 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8c8485648-xl4zk"] Apr 16 20:39:56.970831 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.970815 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:56.973207 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.973188 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Apr 16 20:39:56.973347 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.973332 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:56.973832 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.973804 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Apr 16 20:39:56.973933 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.973889 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Apr 16 20:39:56.973933 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.973894 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r9mb2\"" Apr 16 20:39:56.973933 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.973907 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Apr 16 20:39:56.976340 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.976321 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Apr 16 20:39:56.976442 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.976385 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Apr 16 20:39:56.976501 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.976452 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9vpc9\"" Apr 16 20:39:56.976501 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.976463 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Apr 16 20:39:56.976740 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.976723 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Apr 16 20:39:56.977080 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.977058 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Apr 16 20:39:56.977235 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.977218 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Apr 16 20:39:56.980408 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.980222 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Apr 16 20:39:56.982212 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.982192 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-9d4b6777b-s4bp5"] Apr 16 20:39:56.985205 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:56.985193 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8c8485648-xl4zk"] Apr 16 20:39:57.039118 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039093 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc0a46ee-2e45-4265-8a43-5491b384d9f7-config\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.039210 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039126 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjb4\" (UniqueName: \"kubernetes.io/projected/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-kube-api-access-vwjb4\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:39:57.039210 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039154 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.039210 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039190 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-stats-auth\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.039210 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039206 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7gc\" (UniqueName: \"kubernetes.io/projected/dc0a46ee-2e45-4265-8a43-5491b384d9f7-kube-api-access-ms7gc\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.039355 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039224 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:39:57.039355 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:57.039299 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 20:39:57.039355 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp82p\" (UniqueName: \"kubernetes.io/projected/6721f3c5-b29f-4e7b-989f-9eff82d003b6-kube-api-access-bp82p\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.039355 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:57.039336 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls podName:7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1 nodeName:}" failed. No retries permitted until 2026-04-16 20:39:57.539324023 +0000 UTC m=+127.714688267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-krstf" (UID: "7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1") : secret "samples-operator-tls" not found Apr 16 20:39:57.039355 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039349 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc0a46ee-2e45-4265-8a43-5491b384d9f7-serving-cert\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.039544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039365 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc0a46ee-2e45-4265-8a43-5491b384d9f7-trusted-ca\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.039544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.039544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.039428 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-default-certificate\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.047447 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.047426 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjb4\" (UniqueName: \"kubernetes.io/projected/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-kube-api-access-vwjb4\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:39:57.139844 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.139786 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bp82p\" (UniqueName: \"kubernetes.io/projected/6721f3c5-b29f-4e7b-989f-9eff82d003b6-kube-api-access-bp82p\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.139844 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.139821 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc0a46ee-2e45-4265-8a43-5491b384d9f7-serving-cert\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.139844 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.139838 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc0a46ee-2e45-4265-8a43-5491b384d9f7-trusted-ca\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.140027 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.140000 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.140099 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.140038 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-default-certificate\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.140099 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.140066 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc0a46ee-2e45-4265-8a43-5491b384d9f7-config\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.140202 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.140111 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.140202 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:57.140137 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:39:57.640118694 +0000 UTC m=+127.815482958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : configmap references non-existent config key: service-ca.crt Apr 16 20:39:57.140202 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.140180 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-stats-auth\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.140202 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:57.140186 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 20:39:57.140424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.140208 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7gc\" (UniqueName: \"kubernetes.io/projected/dc0a46ee-2e45-4265-8a43-5491b384d9f7-kube-api-access-ms7gc\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.140424 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:57.140233 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:39:57.64022128 +0000 UTC m=+127.815585524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : secret "router-metrics-certs-default" not found Apr 16 20:39:57.140795 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.140771 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc0a46ee-2e45-4265-8a43-5491b384d9f7-config\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.140909 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.140892 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc0a46ee-2e45-4265-8a43-5491b384d9f7-trusted-ca\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.141978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.141959 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc0a46ee-2e45-4265-8a43-5491b384d9f7-serving-cert\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.142546 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.142524 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-stats-auth\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.142637 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.142622 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-default-certificate\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.148993 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.148975 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp82p\" (UniqueName: \"kubernetes.io/projected/6721f3c5-b29f-4e7b-989f-9eff82d003b6-kube-api-access-bp82p\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.149212 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.149190 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7gc\" (UniqueName: \"kubernetes.io/projected/dc0a46ee-2e45-4265-8a43-5491b384d9f7-kube-api-access-ms7gc\") pod \"console-operator-9d4b6777b-s4bp5\" (UID: \"dc0a46ee-2e45-4265-8a43-5491b384d9f7\") " pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.282465 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.282448 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:39:57.391122 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.391054 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-9d4b6777b-s4bp5"] Apr 16 20:39:57.393852 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:39:57.393817 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0a46ee_2e45_4265_8a43_5491b384d9f7.slice/crio-b6b2006a245bfe1b1180794faeed16bab295083cf1282e1717a3d91752fadc48 WatchSource:0}: Error finding container b6b2006a245bfe1b1180794faeed16bab295083cf1282e1717a3d91752fadc48: Status 404 returned error can't find the container with id b6b2006a245bfe1b1180794faeed16bab295083cf1282e1717a3d91752fadc48 Apr 16 20:39:57.543985 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.543954 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:39:57.544095 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:57.544081 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 20:39:57.544144 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:57.544134 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls podName:7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1 nodeName:}" failed. No retries permitted until 2026-04-16 20:39:58.544119355 +0000 UTC m=+128.719483603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-krstf" (UID: "7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1") : secret "samples-operator-tls" not found Apr 16 20:39:57.644386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.644318 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.644386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.644367 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:57.644516 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:57.644453 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 20:39:57.644516 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:57.644465 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:39:58.64444698 +0000 UTC m=+128.819811228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : configmap references non-existent config key: service-ca.crt Apr 16 20:39:57.644516 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:57.644501 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:39:58.644488144 +0000 UTC m=+128.819852388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : secret "router-metrics-certs-default" not found Apr 16 20:39:57.902318 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:57.902248 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" event={"ID":"dc0a46ee-2e45-4265-8a43-5491b384d9f7","Type":"ContainerStarted","Data":"b6b2006a245bfe1b1180794faeed16bab295083cf1282e1717a3d91752fadc48"} Apr 16 20:39:58.551842 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:58.551806 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:39:58.552010 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:58.551956 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 20:39:58.552064 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:58.552022 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls podName:7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:00.552002264 +0000 UTC m=+130.727366511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-krstf" (UID: "7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1") : secret "samples-operator-tls" not found Apr 16 20:39:58.652620 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:58.652583 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:58.652743 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:58.652691 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:39:58.652800 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:58.652734 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 20:39:58.652842 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:58.652805 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:00.65278481 +0000 UTC m=+130.828149074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : secret "router-metrics-certs-default" not found Apr 16 20:39:58.652842 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:58.652823 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:00.652814212 +0000 UTC m=+130.828178460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : configmap references non-existent config key: service-ca.crt Apr 16 20:39:59.155707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.155674 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:39:59.156105 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:59.155839 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 16 20:39:59.156105 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:39:59.155914 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs podName:b8187ddd-1510-4903-bc28-7206752133c5 nodeName:}" failed. No retries permitted until 2026-04-16 20:42:01.155895053 +0000 UTC m=+251.331259317 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs") pod "network-metrics-daemon-w8xr8" (UID: "b8187ddd-1510-4903-bc28-7206752133c5") : secret "metrics-daemon-secret" not found Apr 16 20:39:59.737093 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.737060 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6"] Apr 16 20:39:59.739795 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.739778 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:39:59.742287 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.742266 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Apr 16 20:39:59.742287 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.742282 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Apr 16 20:39:59.742429 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.742317 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6r8np\"" Apr 16 20:39:59.743499 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.743482 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Apr 16 20:39:59.743554 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.743531 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Apr 16 20:39:59.749005 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.748986 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6"] Apr 16 20:39:59.860841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.860788 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntj9\" (UniqueName: \"kubernetes.io/projected/b064c14c-045d-4f5f-b252-0ddbff4e03db-kube-api-access-qntj9\") pod \"kube-storage-version-migrator-operator-6769c5d45-8xsb6\" (UID: \"b064c14c-045d-4f5f-b252-0ddbff4e03db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:39:59.860841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.860839 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b064c14c-045d-4f5f-b252-0ddbff4e03db-serving-cert\") pod \"kube-storage-version-migrator-operator-6769c5d45-8xsb6\" (UID: \"b064c14c-045d-4f5f-b252-0ddbff4e03db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:39:59.860951 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.860858 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b064c14c-045d-4f5f-b252-0ddbff4e03db-config\") pod \"kube-storage-version-migrator-operator-6769c5d45-8xsb6\" (UID: \"b064c14c-045d-4f5f-b252-0ddbff4e03db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:39:59.906654 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.906633 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/0.log" Apr 16 20:39:59.906733 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.906666 2562 generic.go:358] "Generic (PLEG): container finished" podID="dc0a46ee-2e45-4265-8a43-5491b384d9f7" containerID="7fd29f35ef2d8a0bc29c447e09b6b016a8837d4cdff9226ca50d89e366764ec9" exitCode=255 Apr 16 20:39:59.906733 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.906694 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" event={"ID":"dc0a46ee-2e45-4265-8a43-5491b384d9f7","Type":"ContainerDied","Data":"7fd29f35ef2d8a0bc29c447e09b6b016a8837d4cdff9226ca50d89e366764ec9"} Apr 16 20:39:59.906906 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.906892 2562 scope.go:117] "RemoveContainer" containerID="7fd29f35ef2d8a0bc29c447e09b6b016a8837d4cdff9226ca50d89e366764ec9" Apr 16 20:39:59.961335 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.961313 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qntj9\" (UniqueName: \"kubernetes.io/projected/b064c14c-045d-4f5f-b252-0ddbff4e03db-kube-api-access-qntj9\") pod \"kube-storage-version-migrator-operator-6769c5d45-8xsb6\" (UID: \"b064c14c-045d-4f5f-b252-0ddbff4e03db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:39:59.961425 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.961412 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b064c14c-045d-4f5f-b252-0ddbff4e03db-serving-cert\") pod \"kube-storage-version-migrator-operator-6769c5d45-8xsb6\" (UID: \"b064c14c-045d-4f5f-b252-0ddbff4e03db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:39:59.961489 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.961461 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b064c14c-045d-4f5f-b252-0ddbff4e03db-config\") pod \"kube-storage-version-migrator-operator-6769c5d45-8xsb6\" (UID: \"b064c14c-045d-4f5f-b252-0ddbff4e03db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:39:59.962181 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.962161 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b064c14c-045d-4f5f-b252-0ddbff4e03db-config\") pod \"kube-storage-version-migrator-operator-6769c5d45-8xsb6\" (UID: \"b064c14c-045d-4f5f-b252-0ddbff4e03db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:39:59.963460 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.963441 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b064c14c-045d-4f5f-b252-0ddbff4e03db-serving-cert\") pod \"kube-storage-version-migrator-operator-6769c5d45-8xsb6\" (UID: \"b064c14c-045d-4f5f-b252-0ddbff4e03db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:39:59.968854 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:39:59.968835 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntj9\" (UniqueName: \"kubernetes.io/projected/b064c14c-045d-4f5f-b252-0ddbff4e03db-kube-api-access-qntj9\") pod \"kube-storage-version-migrator-operator-6769c5d45-8xsb6\" (UID: \"b064c14c-045d-4f5f-b252-0ddbff4e03db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:40:00.047994 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.047970 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" Apr 16 20:40:00.157718 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.157682 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6"] Apr 16 20:40:00.161270 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:40:00.161234 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb064c14c_045d_4f5f_b252_0ddbff4e03db.slice/crio-751f1bb2ef3faef5cf72a255805dc092937d86954a8b1bc9b28729fbb3f0da66 WatchSource:0}: Error finding container 751f1bb2ef3faef5cf72a255805dc092937d86954a8b1bc9b28729fbb3f0da66: Status 404 returned error can't find the container with id 751f1bb2ef3faef5cf72a255805dc092937d86954a8b1bc9b28729fbb3f0da66 Apr 16 20:40:00.565540 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.565518 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:40:00.565704 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:00.565687 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 20:40:00.565773 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:00.565762 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls podName:7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:04.565741536 +0000 UTC m=+134.741105795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-krstf" (UID: "7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1") : secret "samples-operator-tls" not found Apr 16 20:40:00.666785 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.666750 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:00.666894 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.666805 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:00.666930 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:00.666908 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:04.666892865 +0000 UTC m=+134.842257209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : configmap references non-existent config key: service-ca.crt Apr 16 20:40:00.666930 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:00.666925 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 20:40:00.666996 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:00.666960 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:04.66694926 +0000 UTC m=+134.842313504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : secret "router-metrics-certs-default" not found Apr 16 20:40:00.909751 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.909658 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" event={"ID":"b064c14c-045d-4f5f-b252-0ddbff4e03db","Type":"ContainerStarted","Data":"751f1bb2ef3faef5cf72a255805dc092937d86954a8b1bc9b28729fbb3f0da66"} Apr 16 20:40:00.911179 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.911154 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/1.log" Apr 16 20:40:00.911617 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.911584 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/0.log" Apr 16 20:40:00.911700 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.911643 2562 generic.go:358] "Generic (PLEG): container finished" podID="dc0a46ee-2e45-4265-8a43-5491b384d9f7" containerID="d7418baec49816271fe484f481d3725a0031d86f78f3ddde3b5c64c64c1460a4" exitCode=255 Apr 16 20:40:00.911700 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.911674 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" event={"ID":"dc0a46ee-2e45-4265-8a43-5491b384d9f7","Type":"ContainerDied","Data":"d7418baec49816271fe484f481d3725a0031d86f78f3ddde3b5c64c64c1460a4"} Apr 16 20:40:00.911810 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.911706 2562 scope.go:117] "RemoveContainer" containerID="7fd29f35ef2d8a0bc29c447e09b6b016a8837d4cdff9226ca50d89e366764ec9" Apr 16 20:40:00.912027 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:00.912007 2562 scope.go:117] "RemoveContainer" containerID="d7418baec49816271fe484f481d3725a0031d86f78f3ddde3b5c64c64c1460a4" Apr 16 20:40:00.912223 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:00.912201 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-9d4b6777b-s4bp5_openshift-console-operator(dc0a46ee-2e45-4265-8a43-5491b384d9f7)\"" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" podUID="dc0a46ee-2e45-4265-8a43-5491b384d9f7" Apr 16 20:40:01.915109 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:01.915082 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/1.log" Apr 16 20:40:01.915548 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:01.915446 2562 scope.go:117] "RemoveContainer" containerID="d7418baec49816271fe484f481d3725a0031d86f78f3ddde3b5c64c64c1460a4" Apr 16 20:40:01.915683 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:01.915663 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-9d4b6777b-s4bp5_openshift-console-operator(dc0a46ee-2e45-4265-8a43-5491b384d9f7)\"" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" podUID="dc0a46ee-2e45-4265-8a43-5491b384d9f7" Apr 16 20:40:02.918384 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:02.918349 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" event={"ID":"b064c14c-045d-4f5f-b252-0ddbff4e03db","Type":"ContainerStarted","Data":"4211cda41c3357da71a1d48204eb0651d116cee3267a2930142aed79ea0328ab"} Apr 16 20:40:02.934461 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:02.934415 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" podStartSLOduration=1.564889429 podStartE2EDuration="3.934398567s" podCreationTimestamp="2026-04-16 20:39:59 +0000 UTC" firstStartedPulling="2026-04-16 20:40:00.163059347 +0000 UTC m=+130.338423591" lastFinishedPulling="2026-04-16 20:40:02.532568472 +0000 UTC m=+132.707932729" observedRunningTime="2026-04-16 20:40:02.934249961 +0000 UTC m=+133.109614228" watchObservedRunningTime="2026-04-16 20:40:02.934398567 +0000 UTC m=+133.109762833" Apr 16 20:40:03.047481 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:03.047453 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jxb6_24dbfe36-1150-4ee2-b906-d2d5107ac31c/dns-node-resolver/0.log" Apr 16 20:40:04.247706 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:04.247676 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qhhws_427d721c-2662-4922-a063-4b44f92e8ca2/node-ca/0.log" Apr 16 20:40:04.591980 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:04.591954 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:40:04.592132 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:04.592111 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 20:40:04.592186 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:04.592178 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls podName:7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:12.592161982 +0000 UTC m=+142.767526226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-krstf" (UID: "7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1") : secret "samples-operator-tls" not found Apr 16 20:40:04.692756 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:04.692729 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:04.692874 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:04.692770 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:04.692874 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:04.692854 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 20:40:04.692948 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:04.692878 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:12.692862557 +0000 UTC m=+142.868226802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : configmap references non-existent config key: service-ca.crt Apr 16 20:40:04.692948 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:04.692898 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:12.692891489 +0000 UTC m=+142.868255732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : secret "router-metrics-certs-default" not found Apr 16 20:40:07.283201 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.283164 2562 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:40:07.283201 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.283206 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:40:07.283647 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.283549 2562 scope.go:117] "RemoveContainer" containerID="d7418baec49816271fe484f481d3725a0031d86f78f3ddde3b5c64c64c1460a4" Apr 16 20:40:07.283747 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:07.283726 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-9d4b6777b-s4bp5_openshift-console-operator(dc0a46ee-2e45-4265-8a43-5491b384d9f7)\"" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" podUID="dc0a46ee-2e45-4265-8a43-5491b384d9f7" Apr 16 20:40:07.894545 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.894511 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f48c97c49-xsfvx"] Apr 16 20:40:07.897353 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.897336 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:07.900848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.900810 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Apr 16 20:40:07.900964 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.900927 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-66wkg\"" Apr 16 20:40:07.901195 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.901180 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Apr 16 20:40:07.901423 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.901411 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Apr 16 20:40:07.906392 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.906373 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Apr 16 20:40:07.909456 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:07.909434 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f48c97c49-xsfvx"] Apr 16 20:40:08.017837 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.017816 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-image-registry-private-configuration\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.017949 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.017847 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6fn9\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-kube-api-access-n6fn9\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.017949 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.017879 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-bound-sa-token\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.018043 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.017949 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.018043 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.018000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-installation-pull-secrets\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.018043 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.018021 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8382ddfe-6641-4918-af16-3a447905ae35-ca-trust-extracted\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.018144 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.018048 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-trusted-ca\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.018144 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.018086 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-registry-certificates\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.118768 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.118748 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-image-registry-private-configuration\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.118850 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.118777 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n6fn9\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-kube-api-access-n6fn9\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.118850 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.118808 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-bound-sa-token\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.118964 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.118859 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.118964 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.118936 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-installation-pull-secrets\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.119076 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.118971 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8382ddfe-6641-4918-af16-3a447905ae35-ca-trust-extracted\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.119076 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:08.119014 2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 20:40:08.119076 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.119020 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-trusted-ca\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.119076 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:08.119032 2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f48c97c49-xsfvx: secret "image-registry-tls" not found Apr 16 20:40:08.119076 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.119055 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-registry-certificates\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.119337 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:08.119086 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls podName:8382ddfe-6641-4918-af16-3a447905ae35 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:08.619064815 +0000 UTC m=+138.794429059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls") pod "image-registry-7f48c97c49-xsfvx" (UID: "8382ddfe-6641-4918-af16-3a447905ae35") : secret "image-registry-tls" not found Apr 16 20:40:08.119450 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.119428 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8382ddfe-6641-4918-af16-3a447905ae35-ca-trust-extracted\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.119715 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.119696 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-registry-certificates\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.120004 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.119983 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-trusted-ca\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.121368 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.121351 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-image-registry-private-configuration\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.121426 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.121409 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-installation-pull-secrets\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.128176 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.128145 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-bound-sa-token\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.128403 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.128382 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6fn9\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-kube-api-access-n6fn9\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.622752 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:08.622714 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:08.623154 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:08.622914 2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 20:40:08.623154 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:08.622935 2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f48c97c49-xsfvx: secret "image-registry-tls" not found Apr 16 20:40:08.623154 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:08.622993 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls podName:8382ddfe-6641-4918-af16-3a447905ae35 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:09.622974161 +0000 UTC m=+139.798338418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls") pod "image-registry-7f48c97c49-xsfvx" (UID: "8382ddfe-6641-4918-af16-3a447905ae35") : secret "image-registry-tls" not found Apr 16 20:40:09.630403 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:09.630352 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:09.630838 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:09.630499 2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 20:40:09.630838 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:09.630520 2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f48c97c49-xsfvx: secret "image-registry-tls" not found Apr 16 20:40:09.630838 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:09.630577 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls podName:8382ddfe-6641-4918-af16-3a447905ae35 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:11.630559845 +0000 UTC m=+141.805924094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls") pod "image-registry-7f48c97c49-xsfvx" (UID: "8382ddfe-6641-4918-af16-3a447905ae35") : secret "image-registry-tls" not found Apr 16 20:40:11.644926 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:11.644884 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:11.645354 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:11.645045 2562 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 20:40:11.645354 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:11.645067 2562 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f48c97c49-xsfvx: secret "image-registry-tls" not found Apr 16 20:40:11.645354 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:11.645126 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls podName:8382ddfe-6641-4918-af16-3a447905ae35 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:15.645110107 +0000 UTC m=+145.820474351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls") pod "image-registry-7f48c97c49-xsfvx" (UID: "8382ddfe-6641-4918-af16-3a447905ae35") : secret "image-registry-tls" not found Apr 16 20:40:12.652709 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:12.652679 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:40:12.653150 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:12.652841 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 20:40:12.653150 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:12.652928 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls podName:7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:28.652905905 +0000 UTC m=+158.828270150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-krstf" (UID: "7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1") : secret "samples-operator-tls" not found Apr 16 20:40:12.753707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:12.753680 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:12.753833 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:12.753746 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:12.753882 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:12.753835 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:28.753817999 +0000 UTC m=+158.929182255 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : configmap references non-existent config key: service-ca.crt Apr 16 20:40:12.753882 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:12.753839 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 20:40:12.753882 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:12.753873 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs podName:6721f3c5-b29f-4e7b-989f-9eff82d003b6 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:28.753863967 +0000 UTC m=+158.929228213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs") pod "router-default-8c8485648-xl4zk" (UID: "6721f3c5-b29f-4e7b-989f-9eff82d003b6") : secret "router-metrics-certs-default" not found Apr 16 20:40:15.676428 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:15.676387 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:15.678673 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:15.678642 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls\") pod \"image-registry-7f48c97c49-xsfvx\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:15.706242 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:15.706218 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:15.828696 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:15.828669 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f48c97c49-xsfvx"] Apr 16 20:40:15.831744 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:40:15.831712 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8382ddfe_6641_4918_af16_3a447905ae35.slice/crio-57ecb5b78ff5e1fcd932d0b8b02a006d4e888c45938292cc57aadb45381b2e7a WatchSource:0}: Error finding container 57ecb5b78ff5e1fcd932d0b8b02a006d4e888c45938292cc57aadb45381b2e7a: Status 404 returned error can't find the container with id 57ecb5b78ff5e1fcd932d0b8b02a006d4e888c45938292cc57aadb45381b2e7a Apr 16 20:40:15.943750 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:15.943682 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" event={"ID":"8382ddfe-6641-4918-af16-3a447905ae35","Type":"ContainerStarted","Data":"5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69"} Apr 16 20:40:15.943750 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:15.943722 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" event={"ID":"8382ddfe-6641-4918-af16-3a447905ae35","Type":"ContainerStarted","Data":"57ecb5b78ff5e1fcd932d0b8b02a006d4e888c45938292cc57aadb45381b2e7a"} Apr 16 20:40:15.943922 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:15.943824 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:15.963749 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:15.963709 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" podStartSLOduration=8.963697223 podStartE2EDuration="8.963697223s" podCreationTimestamp="2026-04-16 20:40:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 20:40:15.963360697 +0000 UTC m=+146.138724963" watchObservedRunningTime="2026-04-16 20:40:15.963697223 +0000 UTC m=+146.139061489" Apr 16 20:40:18.501542 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:18.498356 2562 scope.go:117] "RemoveContainer" containerID="d7418baec49816271fe484f481d3725a0031d86f78f3ddde3b5c64c64c1460a4" Apr 16 20:40:18.952830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:18.952803 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:40:18.953238 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:18.953220 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/1.log" Apr 16 20:40:18.953322 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:18.953264 2562 generic.go:358] "Generic (PLEG): container finished" podID="dc0a46ee-2e45-4265-8a43-5491b384d9f7" containerID="1d00a06a2624cb2208b156271f1ba1bec005b48cf8784c4e54a65576c8d3a0e1" exitCode=255 Apr 16 20:40:18.953374 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:18.953344 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" event={"ID":"dc0a46ee-2e45-4265-8a43-5491b384d9f7","Type":"ContainerDied","Data":"1d00a06a2624cb2208b156271f1ba1bec005b48cf8784c4e54a65576c8d3a0e1"} Apr 16 20:40:18.953429 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:18.953398 2562 scope.go:117] "RemoveContainer" containerID="d7418baec49816271fe484f481d3725a0031d86f78f3ddde3b5c64c64c1460a4" Apr 16 20:40:18.953832 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:18.953813 2562 scope.go:117] "RemoveContainer" containerID="1d00a06a2624cb2208b156271f1ba1bec005b48cf8784c4e54a65576c8d3a0e1" Apr 16 20:40:18.954051 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:18.954028 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-9d4b6777b-s4bp5_openshift-console-operator(dc0a46ee-2e45-4265-8a43-5491b384d9f7)\"" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" podUID="dc0a46ee-2e45-4265-8a43-5491b384d9f7" Apr 16 20:40:19.956867 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:19.956843 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:40:24.717760 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:24.717710 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vxpfw" podUID="c72f51ec-57bd-429f-85d6-aa06c2c66728" Apr 16 20:40:24.721832 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:24.721801 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-qtfdj" podUID="6781237f-45c1-4874-a81d-43042624470f" Apr 16 20:40:24.966446 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:24.966415 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:40:24.966644 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:24.966426 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vxpfw" Apr 16 20:40:26.515328 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:26.515290 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-w8xr8" podUID="b8187ddd-1510-4903-bc28-7206752133c5" Apr 16 20:40:27.283545 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:27.283510 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:40:27.283545 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:27.283547 2562 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:40:27.283887 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:27.283873 2562 scope.go:117] "RemoveContainer" containerID="1d00a06a2624cb2208b156271f1ba1bec005b48cf8784c4e54a65576c8d3a0e1" Apr 16 20:40:27.284054 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:27.284034 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-9d4b6777b-s4bp5_openshift-console-operator(dc0a46ee-2e45-4265-8a43-5491b384d9f7)\"" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" podUID="dc0a46ee-2e45-4265-8a43-5491b384d9f7" Apr 16 20:40:28.668332 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.668283 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:40:28.671070 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.671039 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-krstf\" (UID: \"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:40:28.769506 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.769463 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:28.769671 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.769545 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:28.770169 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.770141 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6721f3c5-b29f-4e7b-989f-9eff82d003b6-service-ca-bundle\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:28.771952 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.771930 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6721f3c5-b29f-4e7b-989f-9eff82d003b6-metrics-certs\") pod \"router-default-8c8485648-xl4zk\" (UID: \"6721f3c5-b29f-4e7b-989f-9eff82d003b6\") " pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:28.787833 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.787808 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:28.910912 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.910880 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8c8485648-xl4zk"] Apr 16 20:40:28.914810 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:40:28.914784 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6721f3c5_b29f_4e7b_989f_9eff82d003b6.slice/crio-c2e72b082a5feacff47674ca2365728eef43afe77519ed39e7b951db034ad141 WatchSource:0}: Error finding container c2e72b082a5feacff47674ca2365728eef43afe77519ed39e7b951db034ad141: Status 404 returned error can't find the container with id c2e72b082a5feacff47674ca2365728eef43afe77519ed39e7b951db034ad141 Apr 16 20:40:28.967964 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.967937 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" Apr 16 20:40:28.983057 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.983001 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8c8485648-xl4zk" event={"ID":"6721f3c5-b29f-4e7b-989f-9eff82d003b6","Type":"ContainerStarted","Data":"cd327ccd63d4986e1cc0cedf55f2617ab966198f4b0aab929f3eccb4ad939778"} Apr 16 20:40:28.983057 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:28.983046 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8c8485648-xl4zk" event={"ID":"6721f3c5-b29f-4e7b-989f-9eff82d003b6","Type":"ContainerStarted","Data":"c2e72b082a5feacff47674ca2365728eef43afe77519ed39e7b951db034ad141"} Apr 16 20:40:29.007850 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.007804 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-8c8485648-xl4zk" podStartSLOduration=33.007789014 podStartE2EDuration="33.007789014s" podCreationTimestamp="2026-04-16 20:39:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 20:40:29.006813303 +0000 UTC m=+159.182177584" watchObservedRunningTime="2026-04-16 20:40:29.007789014 +0000 UTC m=+159.183153280" Apr 16 20:40:29.088291 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.088240 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf"] Apr 16 20:40:29.575571 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.575528 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:40:29.575737 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.575598 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:40:29.577884 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.577857 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c72f51ec-57bd-429f-85d6-aa06c2c66728-metrics-tls\") pod \"dns-default-vxpfw\" (UID: \"c72f51ec-57bd-429f-85d6-aa06c2c66728\") " pod="openshift-dns/dns-default-vxpfw" Apr 16 20:40:29.577994 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.577975 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6781237f-45c1-4874-a81d-43042624470f-cert\") pod \"ingress-canary-qtfdj\" (UID: \"6781237f-45c1-4874-a81d-43042624470f\") " pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:40:29.770724 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.770702 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7vq8v\"" Apr 16 20:40:29.771907 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.771892 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wwh4f\"" Apr 16 20:40:29.778267 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.778255 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qtfdj" Apr 16 20:40:29.778309 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.778281 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vxpfw" Apr 16 20:40:29.789036 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.788996 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:29.791863 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.791835 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:29.920596 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.920569 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qtfdj"] Apr 16 20:40:29.922970 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:40:29.922880 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6781237f_45c1_4874_a81d_43042624470f.slice/crio-d5c01043d9fe772a03fb124d857222a73a7f8707520ae8573ece8766be47f90a WatchSource:0}: Error finding container d5c01043d9fe772a03fb124d857222a73a7f8707520ae8573ece8766be47f90a: Status 404 returned error can't find the container with id d5c01043d9fe772a03fb124d857222a73a7f8707520ae8573ece8766be47f90a Apr 16 20:40:29.947725 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.947696 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vxpfw"] Apr 16 20:40:29.951251 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:40:29.951222 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72f51ec_57bd_429f_85d6_aa06c2c66728.slice/crio-ff29a61a53eb89d8a370788927c8344f303aed2cfa57f329859b9eb517df26d9 WatchSource:0}: Error finding container ff29a61a53eb89d8a370788927c8344f303aed2cfa57f329859b9eb517df26d9: Status 404 returned error can't find the container with id ff29a61a53eb89d8a370788927c8344f303aed2cfa57f329859b9eb517df26d9 Apr 16 20:40:29.986177 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.986155 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vxpfw" event={"ID":"c72f51ec-57bd-429f-85d6-aa06c2c66728","Type":"ContainerStarted","Data":"ff29a61a53eb89d8a370788927c8344f303aed2cfa57f329859b9eb517df26d9"} Apr 16 20:40:29.987133 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.987112 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" event={"ID":"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1","Type":"ContainerStarted","Data":"727d583a56fe0ee5fc0614592ce8e5b25a998a6f360e8be9331fd847ac3728a7"} Apr 16 20:40:29.987964 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.987947 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qtfdj" event={"ID":"6781237f-45c1-4874-a81d-43042624470f","Type":"ContainerStarted","Data":"d5c01043d9fe772a03fb124d857222a73a7f8707520ae8573ece8766be47f90a"} Apr 16 20:40:29.988184 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.988160 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:29.989338 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:29.989320 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8c8485648-xl4zk" Apr 16 20:40:30.097343 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.097279 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-scchg"] Apr 16 20:40:30.100032 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.100014 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.102730 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.102700 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Apr 16 20:40:30.102832 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.102703 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Apr 16 20:40:30.102832 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.102727 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Apr 16 20:40:30.103030 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.103016 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-khb4d\"" Apr 16 20:40:30.103507 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.103482 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Apr 16 20:40:30.124270 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.124252 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-scchg"] Apr 16 20:40:30.156588 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.156568 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f48c97c49-xsfvx"] Apr 16 20:40:30.160747 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.160718 2562 patch_prober.go:28] interesting pod/image-registry-7f48c97c49-xsfvx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Apr 16 20:40:30.160823 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.160777 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" podUID="8382ddfe-6641-4918-af16-3a447905ae35" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 20:40:30.281652 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.281599 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vg4\" (UniqueName: \"kubernetes.io/projected/873664c9-3817-4764-8f81-37ec78380ce2-kube-api-access-k6vg4\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.281855 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.281689 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/873664c9-3817-4764-8f81-37ec78380ce2-data-volume\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.281855 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.281775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/873664c9-3817-4764-8f81-37ec78380ce2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.281855 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.281803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/873664c9-3817-4764-8f81-37ec78380ce2-crio-socket\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.281855 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.281837 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/873664c9-3817-4764-8f81-37ec78380ce2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.382873 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.382806 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/873664c9-3817-4764-8f81-37ec78380ce2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.383003 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.382870 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vg4\" (UniqueName: \"kubernetes.io/projected/873664c9-3817-4764-8f81-37ec78380ce2-kube-api-access-k6vg4\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.383003 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.382915 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/873664c9-3817-4764-8f81-37ec78380ce2-data-volume\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.383003 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.382970 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/873664c9-3817-4764-8f81-37ec78380ce2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.383003 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.383001 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/873664c9-3817-4764-8f81-37ec78380ce2-crio-socket\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.383243 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.383075 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/873664c9-3817-4764-8f81-37ec78380ce2-crio-socket\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.383420 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.383399 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/873664c9-3817-4764-8f81-37ec78380ce2-data-volume\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.383791 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.383762 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/873664c9-3817-4764-8f81-37ec78380ce2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.386143 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.386122 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/873664c9-3817-4764-8f81-37ec78380ce2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.410820 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.410798 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vg4\" (UniqueName: \"kubernetes.io/projected/873664c9-3817-4764-8f81-37ec78380ce2-kube-api-access-k6vg4\") pod \"insights-runtime-extractor-scchg\" (UID: \"873664c9-3817-4764-8f81-37ec78380ce2\") " pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:30.710902 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:30.710825 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-scchg" Apr 16 20:40:32.133676 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:32.133627 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-scchg"] Apr 16 20:40:32.139149 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:40:32.138674 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873664c9_3817_4764_8f81_37ec78380ce2.slice/crio-b920dc54b393f56b3b4433f06412af0c8c9bfa20d444fcc88e6dd9807664b290 WatchSource:0}: Error finding container b920dc54b393f56b3b4433f06412af0c8c9bfa20d444fcc88e6dd9807664b290: Status 404 returned error can't find the container with id b920dc54b393f56b3b4433f06412af0c8c9bfa20d444fcc88e6dd9807664b290 Apr 16 20:40:32.997586 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:32.997504 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qtfdj" event={"ID":"6781237f-45c1-4874-a81d-43042624470f","Type":"ContainerStarted","Data":"f867b2f3503738e92accefa7120c45f0e512ccd9edf8708deb1e112fc1ed6503"} Apr 16 20:40:32.999026 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:32.999002 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vxpfw" event={"ID":"c72f51ec-57bd-429f-85d6-aa06c2c66728","Type":"ContainerStarted","Data":"7421124604dc8f1fa1a9639fbde4245aa98b2f84be6fbaba3338450ea4579b5b"} Apr 16 20:40:32.999026 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:32.999029 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vxpfw" event={"ID":"c72f51ec-57bd-429f-85d6-aa06c2c66728","Type":"ContainerStarted","Data":"6c10a09e238842cf1d9673c9f13d5235eda9ea929d137cd4e0ded9ada6293b1d"} Apr 16 20:40:32.999164 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:32.999083 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vxpfw" Apr 16 20:40:33.000399 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:33.000380 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" event={"ID":"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1","Type":"ContainerStarted","Data":"81cfd78aea056134712bee24aae911b3b2464817c451cdc7ace7265aa137a371"} Apr 16 20:40:33.000495 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:33.000405 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" event={"ID":"7a2ec0d5-0d88-487b-bdd3-1eaf7c33d6b1","Type":"ContainerStarted","Data":"6e6268b12eaa3c9e7fae21ef1dee283bb6fb1a826e725145f500c91dec7a5e46"} Apr 16 20:40:33.001830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:33.001809 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-scchg" event={"ID":"873664c9-3817-4764-8f81-37ec78380ce2","Type":"ContainerStarted","Data":"e2fbe19aaa3f189b5d5face8fc570f174915d89004bf0860b347e5b7a11b9240"} Apr 16 20:40:33.001830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:33.001832 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-scchg" event={"ID":"873664c9-3817-4764-8f81-37ec78380ce2","Type":"ContainerStarted","Data":"c3f16030bb92bbab2361c98bb9880c78d7c9ca474210d35c0ff349341bc864ec"} Apr 16 20:40:33.001948 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:33.001841 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-scchg" event={"ID":"873664c9-3817-4764-8f81-37ec78380ce2","Type":"ContainerStarted","Data":"b920dc54b393f56b3b4433f06412af0c8c9bfa20d444fcc88e6dd9807664b290"} Apr 16 20:40:33.014156 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:33.014111 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qtfdj" podStartSLOduration=129.936216905 podStartE2EDuration="2m12.014100785s" podCreationTimestamp="2026-04-16 20:38:21 +0000 UTC" firstStartedPulling="2026-04-16 20:40:29.925274024 +0000 UTC m=+160.100638270" lastFinishedPulling="2026-04-16 20:40:32.003157892 +0000 UTC m=+162.178522150" observedRunningTime="2026-04-16 20:40:33.01343269 +0000 UTC m=+163.188796957" watchObservedRunningTime="2026-04-16 20:40:33.014100785 +0000 UTC m=+163.189465051" Apr 16 20:40:33.029956 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:33.029911 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-krstf" podStartSLOduration=34.157171209 podStartE2EDuration="37.029896154s" podCreationTimestamp="2026-04-16 20:39:56 +0000 UTC" firstStartedPulling="2026-04-16 20:40:29.126252634 +0000 UTC m=+159.301616879" lastFinishedPulling="2026-04-16 20:40:31.99897758 +0000 UTC m=+162.174341824" observedRunningTime="2026-04-16 20:40:33.028660662 +0000 UTC m=+163.204024930" watchObservedRunningTime="2026-04-16 20:40:33.029896154 +0000 UTC m=+163.205260420" Apr 16 20:40:33.046086 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:33.046038 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-vxpfw" podStartSLOduration=130.001159902 podStartE2EDuration="2m12.04602478s" podCreationTimestamp="2026-04-16 20:38:21 +0000 UTC" firstStartedPulling="2026-04-16 20:40:29.953496268 +0000 UTC m=+160.128860513" lastFinishedPulling="2026-04-16 20:40:31.998361135 +0000 UTC m=+162.173725391" observedRunningTime="2026-04-16 20:40:33.045437364 +0000 UTC m=+163.220801643" watchObservedRunningTime="2026-04-16 20:40:33.04602478 +0000 UTC m=+163.221389048" Apr 16 20:40:35.007901 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:35.007867 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-scchg" event={"ID":"873664c9-3817-4764-8f81-37ec78380ce2","Type":"ContainerStarted","Data":"99f7b44144956125b407faba2403748e11141a45dcf5ad7284fe5322b1a94e4a"} Apr 16 20:40:39.497514 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:39.497427 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:40:40.160905 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:40.160870 2562 patch_prober.go:28] interesting pod/image-registry-7f48c97c49-xsfvx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Apr 16 20:40:40.161048 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:40.160923 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" podUID="8382ddfe-6641-4918-af16-3a447905ae35" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 20:40:40.498521 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:40.498452 2562 scope.go:117] "RemoveContainer" containerID="1d00a06a2624cb2208b156271f1ba1bec005b48cf8784c4e54a65576c8d3a0e1" Apr 16 20:40:41.024096 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.024073 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:40:41.024246 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.024162 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" event={"ID":"dc0a46ee-2e45-4265-8a43-5491b384d9f7","Type":"ContainerStarted","Data":"ed721d950e8426a4ea2f0cc5c5114306295c3fecfeb67ac2db387087096f7bb1"} Apr 16 20:40:41.024481 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.024458 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:40:41.064703 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.064663 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" podStartSLOduration=42.889331974 podStartE2EDuration="45.064648282s" podCreationTimestamp="2026-04-16 20:39:56 +0000 UTC" firstStartedPulling="2026-04-16 20:39:57.395692676 +0000 UTC m=+127.571056921" lastFinishedPulling="2026-04-16 20:39:59.571008982 +0000 UTC m=+129.746373229" observedRunningTime="2026-04-16 20:40:41.064133196 +0000 UTC m=+171.239497463" watchObservedRunningTime="2026-04-16 20:40:41.064648282 +0000 UTC m=+171.240012547" Apr 16 20:40:41.064997 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.064974 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-scchg" podStartSLOduration=9.284245235 podStartE2EDuration="11.064967364s" podCreationTimestamp="2026-04-16 20:40:30 +0000 UTC" firstStartedPulling="2026-04-16 20:40:32.212812179 +0000 UTC m=+162.388176427" lastFinishedPulling="2026-04-16 20:40:33.993534299 +0000 UTC m=+164.168898556" observedRunningTime="2026-04-16 20:40:35.027794044 +0000 UTC m=+165.203158310" watchObservedRunningTime="2026-04-16 20:40:41.064967364 +0000 UTC m=+171.240331631" Apr 16 20:40:41.081925 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.081904 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw"] Apr 16 20:40:41.085018 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.085004 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.088996 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.088965 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Apr 16 20:40:41.089138 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.088971 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Apr 16 20:40:41.089138 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.089022 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Apr 16 20:40:41.089138 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.089027 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n6b47\"" Apr 16 20:40:41.089332 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.089320 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Apr 16 20:40:41.089464 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.089450 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Apr 16 20:40:41.099981 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.099961 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw"] Apr 16 20:40:41.146516 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.146494 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cgclg"] Apr 16 20:40:41.149335 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.149321 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.154545 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.154525 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Apr 16 20:40:41.154669 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.154562 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Apr 16 20:40:41.154669 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.154525 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fn5z7\"" Apr 16 20:40:41.154669 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.154658 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Apr 16 20:40:41.197132 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.197112 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-9d4b6777b-s4bp5" Apr 16 20:40:41.255385 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-tls\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.255385 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.255385 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255249 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.255385 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255289 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b5670c6-cfdc-4985-925a-8ab32521a7ce-root\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.255385 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvf8\" (UniqueName: \"kubernetes.io/projected/8b5670c6-cfdc-4985-925a-8ab32521a7ce-kube-api-access-kkvf8\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.255805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255449 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b5670c6-cfdc-4985-925a-8ab32521a7ce-metrics-client-ca\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.255805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255504 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.255805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255535 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-wtmp\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.255805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwn5f\" (UniqueName: \"kubernetes.io/projected/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-kube-api-access-rwn5f\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.255805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.255805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-metrics-client-ca\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.255805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-textfile\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.255805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.255740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b5670c6-cfdc-4985-925a-8ab32521a7ce-sys\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.356677 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b5670c6-cfdc-4985-925a-8ab32521a7ce-sys\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.356724 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-tls\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.356758 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.356804 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.356836 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b5670c6-cfdc-4985-925a-8ab32521a7ce-root\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.356859 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvf8\" (UniqueName: \"kubernetes.io/projected/8b5670c6-cfdc-4985-925a-8ab32521a7ce-kube-api-access-kkvf8\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.356886 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b5670c6-cfdc-4985-925a-8ab32521a7ce-metrics-client-ca\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.356922 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.356946 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-wtmp\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.356987 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rwn5f\" (UniqueName: \"kubernetes.io/projected/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-kube-api-access-rwn5f\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.357015 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.357042 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-metrics-client-ca\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.357064 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-textfile\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.357416 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-textfile\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.359936 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.357483 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b5670c6-cfdc-4985-925a-8ab32521a7ce-sys\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.360785 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:41.360279 2562 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Apr 16 20:40:41.360785 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:41.360364 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-openshift-state-metrics-tls podName:47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116 nodeName:}" failed. No retries permitted until 2026-04-16 20:40:41.860341879 +0000 UTC m=+172.035706126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-openshift-state-metrics-tls") pod "openshift-state-metrics-9d44df66c-m48sw" (UID: "47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116") : secret "openshift-state-metrics-tls" not found Apr 16 20:40:41.360785 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.360740 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-wtmp\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.361363 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.361340 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b5670c6-cfdc-4985-925a-8ab32521a7ce-root\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.361694 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.361668 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-tls\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.361998 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.361977 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.362453 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.362418 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b5670c6-cfdc-4985-925a-8ab32521a7ce-metrics-client-ca\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.362453 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.362423 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-metrics-client-ca\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.362590 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.362488 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b5670c6-cfdc-4985-925a-8ab32521a7ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.364549 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.364528 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.389290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.389272 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvf8\" (UniqueName: \"kubernetes.io/projected/8b5670c6-cfdc-4985-925a-8ab32521a7ce-kube-api-access-kkvf8\") pod \"node-exporter-cgclg\" (UID: \"8b5670c6-cfdc-4985-925a-8ab32521a7ce\") " pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.391379 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.391357 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwn5f\" (UniqueName: \"kubernetes.io/projected/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-kube-api-access-rwn5f\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.457547 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.457520 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cgclg" Apr 16 20:40:41.465229 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:40:41.465206 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5670c6_cfdc_4985_925a_8ab32521a7ce.slice/crio-a065e540e043f6ffd4168ad38d4ef3a26037ab58702d5dbbf6395b8eceba3e35 WatchSource:0}: Error finding container a065e540e043f6ffd4168ad38d4ef3a26037ab58702d5dbbf6395b8eceba3e35: Status 404 returned error can't find the container with id a065e540e043f6ffd4168ad38d4ef3a26037ab58702d5dbbf6395b8eceba3e35 Apr 16 20:40:41.861001 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.860972 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.863549 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.863526 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-m48sw\" (UID: \"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:41.995228 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:41.995192 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" Apr 16 20:40:42.029391 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:42.029294 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cgclg" event={"ID":"8b5670c6-cfdc-4985-925a-8ab32521a7ce","Type":"ContainerStarted","Data":"a065e540e043f6ffd4168ad38d4ef3a26037ab58702d5dbbf6395b8eceba3e35"} Apr 16 20:40:42.283467 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:42.283442 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw"] Apr 16 20:40:42.288328 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:40:42.288294 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f2c6c8_a0c6_4fb4_8cb8_ac5b5d7e6116.slice/crio-3044e76e7987e94122923f7d95a971aa1e3311ba5b9698f5bf8881e5c4e6b8df WatchSource:0}: Error finding container 3044e76e7987e94122923f7d95a971aa1e3311ba5b9698f5bf8881e5c4e6b8df: Status 404 returned error can't find the container with id 3044e76e7987e94122923f7d95a971aa1e3311ba5b9698f5bf8881e5c4e6b8df Apr 16 20:40:43.006431 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:43.006393 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vxpfw" Apr 16 20:40:43.033440 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:43.033358 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" event={"ID":"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116","Type":"ContainerStarted","Data":"f641c5d1d13411cfc8368d081fc3b616efcacb377e9cab8dd75f2bc0cb82f836"} Apr 16 20:40:43.033440 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:43.033402 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" event={"ID":"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116","Type":"ContainerStarted","Data":"6923542ac298fbd0add583abbb4951059de52c3e0ff4b12239b0519811b47f8f"} Apr 16 20:40:43.033440 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:43.033420 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" event={"ID":"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116","Type":"ContainerStarted","Data":"3044e76e7987e94122923f7d95a971aa1e3311ba5b9698f5bf8881e5c4e6b8df"} Apr 16 20:40:43.035291 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:43.035252 2562 generic.go:358] "Generic (PLEG): container finished" podID="8b5670c6-cfdc-4985-925a-8ab32521a7ce" containerID="62bc9a36be489b087d0a40ce7f1dc573e0b0bf86dd1070a26d1fd118b553a5ba" exitCode=0 Apr 16 20:40:43.035385 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:43.035311 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cgclg" event={"ID":"8b5670c6-cfdc-4985-925a-8ab32521a7ce","Type":"ContainerDied","Data":"62bc9a36be489b087d0a40ce7f1dc573e0b0bf86dd1070a26d1fd118b553a5ba"} Apr 16 20:40:44.040075 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:44.040043 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" event={"ID":"47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116","Type":"ContainerStarted","Data":"3387889bec7bc2b7a4727b52d959d5eec67377cf25e26ed3bf9e7600e1322a24"} Apr 16 20:40:44.041818 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:44.041795 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cgclg" event={"ID":"8b5670c6-cfdc-4985-925a-8ab32521a7ce","Type":"ContainerStarted","Data":"224926b76a72aa3d8475bb4b158ca06518307f704570afa976d321da2b8202f1"} Apr 16 20:40:44.041922 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:44.041826 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cgclg" event={"ID":"8b5670c6-cfdc-4985-925a-8ab32521a7ce","Type":"ContainerStarted","Data":"f1a94d9a63e635249592b7d6cf124e120d95297745fc110626cab0973a93f5be"} Apr 16 20:40:44.059183 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:44.059139 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-m48sw" podStartSLOduration=2.074349501 podStartE2EDuration="3.059127595s" podCreationTimestamp="2026-04-16 20:40:41 +0000 UTC" firstStartedPulling="2026-04-16 20:40:42.442317048 +0000 UTC m=+172.617681292" lastFinishedPulling="2026-04-16 20:40:43.427095138 +0000 UTC m=+173.602459386" observedRunningTime="2026-04-16 20:40:44.058731745 +0000 UTC m=+174.234096012" watchObservedRunningTime="2026-04-16 20:40:44.059127595 +0000 UTC m=+174.234491861" Apr 16 20:40:44.082134 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:44.082094 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cgclg" podStartSLOduration=2.358054132 podStartE2EDuration="3.082082876s" podCreationTimestamp="2026-04-16 20:40:41 +0000 UTC" firstStartedPulling="2026-04-16 20:40:41.466805838 +0000 UTC m=+171.642170082" lastFinishedPulling="2026-04-16 20:40:42.190834568 +0000 UTC m=+172.366198826" observedRunningTime="2026-04-16 20:40:44.080865706 +0000 UTC m=+174.256229971" watchObservedRunningTime="2026-04-16 20:40:44.082082876 +0000 UTC m=+174.257447142" Apr 16 20:40:45.649931 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.649905 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n"] Apr 16 20:40:45.653193 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.653173 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.655804 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.655784 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Apr 16 20:40:45.655901 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.655786 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Apr 16 20:40:45.657312 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.657291 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Apr 16 20:40:45.657396 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.657368 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3prtbro9kajuf\"" Apr 16 20:40:45.657396 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.657303 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Apr 16 20:40:45.657512 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.657499 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fmv5g\"" Apr 16 20:40:45.665360 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.665339 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n"] Apr 16 20:40:45.688187 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.688161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2e40448a-e313-48bc-9a04-1475811e4924-client-ca-bundle\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.688268 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.688189 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2e40448a-e313-48bc-9a04-1475811e4924-secret-metrics-server-tls\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.688268 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.688208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2e40448a-e313-48bc-9a04-1475811e4924-metrics-server-audit-profiles\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.688268 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.688226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt5s6\" (UniqueName: \"kubernetes.io/projected/2e40448a-e313-48bc-9a04-1475811e4924-kube-api-access-kt5s6\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.688364 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.688294 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2e40448a-e313-48bc-9a04-1475811e4924-secret-metrics-server-client-certs\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.688364 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.688326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e40448a-e313-48bc-9a04-1475811e4924-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.688364 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.688355 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2e40448a-e313-48bc-9a04-1475811e4924-audit-log\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.788829 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.788805 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2e40448a-e313-48bc-9a04-1475811e4924-secret-metrics-server-client-certs\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.788952 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.788845 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e40448a-e313-48bc-9a04-1475811e4924-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.788952 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.788884 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2e40448a-e313-48bc-9a04-1475811e4924-audit-log\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.788952 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.788911 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2e40448a-e313-48bc-9a04-1475811e4924-client-ca-bundle\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.788952 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.788938 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2e40448a-e313-48bc-9a04-1475811e4924-secret-metrics-server-tls\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.789211 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.789186 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2e40448a-e313-48bc-9a04-1475811e4924-metrics-server-audit-profiles\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.789276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.789233 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt5s6\" (UniqueName: \"kubernetes.io/projected/2e40448a-e313-48bc-9a04-1475811e4924-kube-api-access-kt5s6\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.789723 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.789679 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2e40448a-e313-48bc-9a04-1475811e4924-audit-log\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.790445 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.790419 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2e40448a-e313-48bc-9a04-1475811e4924-metrics-server-audit-profiles\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.790583 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.790562 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e40448a-e313-48bc-9a04-1475811e4924-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.791456 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.791423 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2e40448a-e313-48bc-9a04-1475811e4924-secret-metrics-server-tls\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.791653 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.791635 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2e40448a-e313-48bc-9a04-1475811e4924-secret-metrics-server-client-certs\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.791719 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.791668 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2e40448a-e313-48bc-9a04-1475811e4924-client-ca-bundle\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.800474 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.800454 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt5s6\" (UniqueName: \"kubernetes.io/projected/2e40448a-e313-48bc-9a04-1475811e4924-kube-api-access-kt5s6\") pod \"metrics-server-7f6fb6ccd-9tk8n\" (UID: \"2e40448a-e313-48bc-9a04-1475811e4924\") " pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:45.962886 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:45.962827 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:40:46.085783 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:46.085751 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n"] Apr 16 20:40:46.088554 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:40:46.088529 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e40448a_e313_48bc_9a04_1475811e4924.slice/crio-87fcda0777721275932fa92ccc0e028d789000467251a11d02efa8da041c2cb4 WatchSource:0}: Error finding container 87fcda0777721275932fa92ccc0e028d789000467251a11d02efa8da041c2cb4: Status 404 returned error can't find the container with id 87fcda0777721275932fa92ccc0e028d789000467251a11d02efa8da041c2cb4 Apr 16 20:40:47.050940 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:47.050894 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" event={"ID":"2e40448a-e313-48bc-9a04-1475811e4924","Type":"ContainerStarted","Data":"87fcda0777721275932fa92ccc0e028d789000467251a11d02efa8da041c2cb4"} Apr 16 20:40:48.054957 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:48.054924 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" event={"ID":"2e40448a-e313-48bc-9a04-1475811e4924","Type":"ContainerStarted","Data":"ccdd4fadd291701d3eda2b627fc53a62e52754b1b5ab686a389ca23d62fe98c0"} Apr 16 20:40:48.077003 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:48.076962 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" podStartSLOduration=1.5239876940000001 podStartE2EDuration="3.076948592s" podCreationTimestamp="2026-04-16 20:40:45 +0000 UTC" firstStartedPulling="2026-04-16 20:40:46.090368016 +0000 UTC m=+176.265732260" lastFinishedPulling="2026-04-16 20:40:47.643328909 +0000 UTC m=+177.818693158" observedRunningTime="2026-04-16 20:40:48.073253853 +0000 UTC m=+178.248618119" watchObservedRunningTime="2026-04-16 20:40:48.076948592 +0000 UTC m=+178.252312897" Apr 16 20:40:50.161088 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:50.161061 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:55.175010 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.174948 2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" podUID="8382ddfe-6641-4918-af16-3a447905ae35" containerName="registry" containerID="cri-o://5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69" gracePeriod=30 Apr 16 20:40:55.407981 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.407959 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:55.449720 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.449667 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-trusted-ca\") pod \"8382ddfe-6641-4918-af16-3a447905ae35\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " Apr 16 20:40:55.449720 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.449698 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls\") pod \"8382ddfe-6641-4918-af16-3a447905ae35\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " Apr 16 20:40:55.449873 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.449747 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-registry-certificates\") pod \"8382ddfe-6641-4918-af16-3a447905ae35\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " Apr 16 20:40:55.449873 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.449768 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-image-registry-private-configuration\") pod \"8382ddfe-6641-4918-af16-3a447905ae35\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " Apr 16 20:40:55.449873 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.449789 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6fn9\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-kube-api-access-n6fn9\") pod \"8382ddfe-6641-4918-af16-3a447905ae35\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " Apr 16 20:40:55.449873 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.449834 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8382ddfe-6641-4918-af16-3a447905ae35-ca-trust-extracted\") pod \"8382ddfe-6641-4918-af16-3a447905ae35\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " Apr 16 20:40:55.449873 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.449873 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-bound-sa-token\") pod \"8382ddfe-6641-4918-af16-3a447905ae35\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " Apr 16 20:40:55.450091 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.449895 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-installation-pull-secrets\") pod \"8382ddfe-6641-4918-af16-3a447905ae35\" (UID: \"8382ddfe-6641-4918-af16-3a447905ae35\") " Apr 16 20:40:55.453032 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.450714 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8382ddfe-6641-4918-af16-3a447905ae35" (UID: "8382ddfe-6641-4918-af16-3a447905ae35"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:40:55.453032 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.450927 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8382ddfe-6641-4918-af16-3a447905ae35" (UID: "8382ddfe-6641-4918-af16-3a447905ae35"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:40:55.453032 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.451089 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-trusted-ca\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:40:55.453392 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.453366 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8382ddfe-6641-4918-af16-3a447905ae35" (UID: "8382ddfe-6641-4918-af16-3a447905ae35"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:40:55.453471 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.453366 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8382ddfe-6641-4918-af16-3a447905ae35" (UID: "8382ddfe-6641-4918-af16-3a447905ae35"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:40:55.453529 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.453493 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8382ddfe-6641-4918-af16-3a447905ae35-registry-certificates\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:40:55.455659 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.453855 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-kube-api-access-n6fn9" (OuterVolumeSpecName: "kube-api-access-n6fn9") pod "8382ddfe-6641-4918-af16-3a447905ae35" (UID: "8382ddfe-6641-4918-af16-3a447905ae35"). InnerVolumeSpecName "kube-api-access-n6fn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:40:55.458106 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.458054 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8382ddfe-6641-4918-af16-3a447905ae35" (UID: "8382ddfe-6641-4918-af16-3a447905ae35"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 20:40:55.458957 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.458661 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8382ddfe-6641-4918-af16-3a447905ae35" (UID: "8382ddfe-6641-4918-af16-3a447905ae35"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 20:40:55.463765 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.463741 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382ddfe-6641-4918-af16-3a447905ae35-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8382ddfe-6641-4918-af16-3a447905ae35" (UID: "8382ddfe-6641-4918-af16-3a447905ae35"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:40:55.554109 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.554086 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8382ddfe-6641-4918-af16-3a447905ae35-ca-trust-extracted\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:40:55.554109 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.554106 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-bound-sa-token\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:40:55.554222 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.554116 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-installation-pull-secrets\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:40:55.554222 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.554125 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-registry-tls\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:40:55.554222 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.554135 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8382ddfe-6641-4918-af16-3a447905ae35-image-registry-private-configuration\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:40:55.554222 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:55.554144 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6fn9\" (UniqueName: \"kubernetes.io/projected/8382ddfe-6641-4918-af16-3a447905ae35-kube-api-access-n6fn9\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:40:56.076038 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:56.076010 2562 generic.go:358] "Generic (PLEG): container finished" podID="8382ddfe-6641-4918-af16-3a447905ae35" containerID="5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69" exitCode=0 Apr 16 20:40:56.076151 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:56.076101 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" event={"ID":"8382ddfe-6641-4918-af16-3a447905ae35","Type":"ContainerDied","Data":"5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69"} Apr 16 20:40:56.076151 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:56.076127 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" Apr 16 20:40:56.076151 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:56.076144 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f48c97c49-xsfvx" event={"ID":"8382ddfe-6641-4918-af16-3a447905ae35","Type":"ContainerDied","Data":"57ecb5b78ff5e1fcd932d0b8b02a006d4e888c45938292cc57aadb45381b2e7a"} Apr 16 20:40:56.076244 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:56.076163 2562 scope.go:117] "RemoveContainer" containerID="5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69" Apr 16 20:40:56.083808 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:56.083672 2562 scope.go:117] "RemoveContainer" containerID="5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69" Apr 16 20:40:56.083983 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:40:56.083958 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69\": container with ID starting with 5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69 not found: ID does not exist" containerID="5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69" Apr 16 20:40:56.084041 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:56.083995 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69"} err="failed to get container status \"5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69\": rpc error: code = NotFound desc = could not find container \"5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69\": container with ID starting with 5dc98a086991bac3a4bd9c9277d1b3884aa465a253f87d8abbb6ed1feec0dd69 not found: ID does not exist" Apr 16 20:40:56.097752 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:56.097731 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f48c97c49-xsfvx"] Apr 16 20:40:56.101727 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:56.101709 2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f48c97c49-xsfvx"] Apr 16 20:40:56.501392 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:56.501322 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8382ddfe-6641-4918-af16-3a447905ae35" path="/var/lib/kubelet/pods/8382ddfe-6641-4918-af16-3a447905ae35/volumes" Apr 16 20:40:57.110651 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.110619 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-c7b887fd4-zx2xn"] Apr 16 20:40:57.110879 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.110867 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8382ddfe-6641-4918-af16-3a447905ae35" containerName="registry" Apr 16 20:40:57.110923 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.110881 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8382ddfe-6641-4918-af16-3a447905ae35" containerName="registry" Apr 16 20:40:57.110960 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.110926 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8382ddfe-6641-4918-af16-3a447905ae35" containerName="registry" Apr 16 20:40:57.115557 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.115541 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.118362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.118338 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pdmrs\"" Apr 16 20:40:57.118488 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.118341 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Apr 16 20:40:57.119545 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.119529 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Apr 16 20:40:57.120106 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.119976 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Apr 16 20:40:57.120106 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.120019 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Apr 16 20:40:57.120106 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.120032 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Apr 16 20:40:57.120106 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.119979 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Apr 16 20:40:57.120293 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.120035 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Apr 16 20:40:57.123995 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.123975 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7b887fd4-zx2xn"] Apr 16 20:40:57.125326 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.125308 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Apr 16 20:40:57.165733 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.165707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-trusted-ca-bundle\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.165880 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.165742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-serving-cert\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.165880 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.165761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-service-ca\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.165880 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.165786 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-console-config\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.165880 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.165822 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-oauth-serving-cert\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.165880 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.165864 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-oauth-config\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.166042 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.165888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8m27\" (UniqueName: \"kubernetes.io/projected/381b6539-2390-41f0-b3b1-50783e6e6649-kube-api-access-t8m27\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.266838 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.266811 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t8m27\" (UniqueName: \"kubernetes.io/projected/381b6539-2390-41f0-b3b1-50783e6e6649-kube-api-access-t8m27\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.266948 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.266873 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-trusted-ca-bundle\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.266948 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.266901 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-serving-cert\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.266948 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.266926 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-service-ca\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.266948 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.266944 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-console-config\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.267119 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.266960 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-oauth-serving-cert\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.267119 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.266982 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-oauth-config\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.267830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.267802 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-service-ca\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.267913 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.267887 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-console-config\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.268040 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.268023 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-trusted-ca-bundle\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.268407 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.268389 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-oauth-serving-cert\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.269513 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.269485 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-oauth-config\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.269628 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.269536 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-serving-cert\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.275840 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.275823 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8m27\" (UniqueName: \"kubernetes.io/projected/381b6539-2390-41f0-b3b1-50783e6e6649-kube-api-access-t8m27\") pod \"console-c7b887fd4-zx2xn\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.425895 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.425817 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:40:57.541025 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:57.540951 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7b887fd4-zx2xn"] Apr 16 20:40:57.546009 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:40:57.545979 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381b6539_2390_41f0_b3b1_50783e6e6649.slice/crio-4df49671682578c4cf114c42a54ad2924fbde81d171b7139ad179a5f4c60c116 WatchSource:0}: Error finding container 4df49671682578c4cf114c42a54ad2924fbde81d171b7139ad179a5f4c60c116: Status 404 returned error can't find the container with id 4df49671682578c4cf114c42a54ad2924fbde81d171b7139ad179a5f4c60c116 Apr 16 20:40:58.083383 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:40:58.083353 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7b887fd4-zx2xn" event={"ID":"381b6539-2390-41f0-b3b1-50783e6e6649","Type":"ContainerStarted","Data":"4df49671682578c4cf114c42a54ad2924fbde81d171b7139ad179a5f4c60c116"} Apr 16 20:41:01.092544 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:01.092511 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7b887fd4-zx2xn" event={"ID":"381b6539-2390-41f0-b3b1-50783e6e6649","Type":"ContainerStarted","Data":"0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d"} Apr 16 20:41:01.117243 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:01.117198 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-c7b887fd4-zx2xn" podStartSLOduration=1.408967874 podStartE2EDuration="4.117185108s" podCreationTimestamp="2026-04-16 20:40:57 +0000 UTC" firstStartedPulling="2026-04-16 20:40:57.547775576 +0000 UTC m=+187.723139820" lastFinishedPulling="2026-04-16 20:41:00.255992807 +0000 UTC m=+190.431357054" observedRunningTime="2026-04-16 20:41:01.11558001 +0000 UTC m=+191.290944276" watchObservedRunningTime="2026-04-16 20:41:01.117185108 +0000 UTC m=+191.292549374" Apr 16 20:41:05.963862 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:05.963823 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:41:05.963862 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:05.963862 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:41:07.426012 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:07.425980 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:41:07.426012 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:07.426014 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:41:07.430350 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:07.430327 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:41:08.114683 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:08.114657 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:41:13.125074 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:13.125046 2562 generic.go:358] "Generic (PLEG): container finished" podID="b064c14c-045d-4f5f-b252-0ddbff4e03db" containerID="4211cda41c3357da71a1d48204eb0651d116cee3267a2930142aed79ea0328ab" exitCode=0 Apr 16 20:41:13.125379 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:13.125119 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" event={"ID":"b064c14c-045d-4f5f-b252-0ddbff4e03db","Type":"ContainerDied","Data":"4211cda41c3357da71a1d48204eb0651d116cee3267a2930142aed79ea0328ab"} Apr 16 20:41:13.125435 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:13.125422 2562 scope.go:117] "RemoveContainer" containerID="4211cda41c3357da71a1d48204eb0651d116cee3267a2930142aed79ea0328ab" Apr 16 20:41:14.129511 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:14.129477 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6769c5d45-8xsb6" event={"ID":"b064c14c-045d-4f5f-b252-0ddbff4e03db","Type":"ContainerStarted","Data":"b2dbd43379005e59d42a7aa79a316e63952d4f1646dc33ca7c55791119b3df55"} Apr 16 20:41:25.969126 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:25.969097 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:41:25.972895 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:41:25.972870 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f6fb6ccd-9tk8n" Apr 16 20:42:01.210519 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:01.210476 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:42:01.212685 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:01.212666 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8187ddd-1510-4903-bc28-7206752133c5-metrics-certs\") pod \"network-metrics-daemon-w8xr8\" (UID: \"b8187ddd-1510-4903-bc28-7206752133c5\") " pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:42:01.401058 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:01.401029 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vdj2n\"" Apr 16 20:42:01.409436 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:01.409408 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8xr8" Apr 16 20:42:01.526640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:01.526568 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w8xr8"] Apr 16 20:42:01.529300 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:42:01.529273 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8187ddd_1510_4903_bc28_7206752133c5.slice/crio-2341d8bd62d976e119738964ac0fb9553b652c680262e78942a030b122282a8a WatchSource:0}: Error finding container 2341d8bd62d976e119738964ac0fb9553b652c680262e78942a030b122282a8a: Status 404 returned error can't find the container with id 2341d8bd62d976e119738964ac0fb9553b652c680262e78942a030b122282a8a Apr 16 20:42:02.268958 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:02.268922 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8xr8" event={"ID":"b8187ddd-1510-4903-bc28-7206752133c5","Type":"ContainerStarted","Data":"2341d8bd62d976e119738964ac0fb9553b652c680262e78942a030b122282a8a"} Apr 16 20:42:03.273262 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:03.273231 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8xr8" event={"ID":"b8187ddd-1510-4903-bc28-7206752133c5","Type":"ContainerStarted","Data":"8e1213044ff3372b4890899c88d0061ce598ed6c3d4a1b0f27d0a43f4e2752ce"} Apr 16 20:42:03.273640 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:03.273269 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8xr8" event={"ID":"b8187ddd-1510-4903-bc28-7206752133c5","Type":"ContainerStarted","Data":"de35ad4c6304d67a5bc9b9e8439799f19858e7c22b4f97b732e02547b3280985"} Apr 16 20:42:03.289115 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:03.289079 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w8xr8" podStartSLOduration=252.252878138 podStartE2EDuration="4m13.289069108s" podCreationTimestamp="2026-04-16 20:37:50 +0000 UTC" firstStartedPulling="2026-04-16 20:42:01.53121377 +0000 UTC m=+251.706578014" lastFinishedPulling="2026-04-16 20:42:02.567404736 +0000 UTC m=+252.742768984" observedRunningTime="2026-04-16 20:42:03.288624062 +0000 UTC m=+253.463988331" watchObservedRunningTime="2026-04-16 20:42:03.289069108 +0000 UTC m=+253.464433373" Apr 16 20:42:10.299237 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.299166 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7754d9dc6b-v97xj"] Apr 16 20:42:10.301213 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.301183 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.315358 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.315336 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7754d9dc6b-v97xj"] Apr 16 20:42:10.374269 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.374244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-oauth-serving-cert\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.374395 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.374281 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-trusted-ca-bundle\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.374395 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.374300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-serving-cert\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.374515 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.374414 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-oauth-config\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.374515 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.374472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-service-ca\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.374515 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.374505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-config\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.374657 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.374536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nr8q\" (UniqueName: \"kubernetes.io/projected/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-kube-api-access-7nr8q\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.475289 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.475263 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-oauth-serving-cert\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.475407 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.475296 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-trusted-ca-bundle\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.475407 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.475316 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-serving-cert\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.475407 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.475345 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-oauth-config\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.475563 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.475457 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-service-ca\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.475563 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.475501 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-config\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.475563 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.475532 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7nr8q\" (UniqueName: \"kubernetes.io/projected/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-kube-api-access-7nr8q\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.476090 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.476061 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-oauth-serving-cert\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.476220 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.476200 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-config\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.476220 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.476210 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-service-ca\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.476342 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.476288 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-trusted-ca-bundle\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.477841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.477815 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-serving-cert\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.477841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.477823 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-oauth-config\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.485014 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.484990 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nr8q\" (UniqueName: \"kubernetes.io/projected/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-kube-api-access-7nr8q\") pod \"console-7754d9dc6b-v97xj\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.612081 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.612025 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:10.736517 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:10.736254 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7754d9dc6b-v97xj"] Apr 16 20:42:10.739416 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:42:10.739386 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987fefec_8d26_4f7b_b9b2_e2cc90a63d2b.slice/crio-4b89e31779d7dcb12e9a3626376e1856b55286df6a5c79284705cd135349e4f3 WatchSource:0}: Error finding container 4b89e31779d7dcb12e9a3626376e1856b55286df6a5c79284705cd135349e4f3: Status 404 returned error can't find the container with id 4b89e31779d7dcb12e9a3626376e1856b55286df6a5c79284705cd135349e4f3 Apr 16 20:42:11.296816 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:11.296781 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7754d9dc6b-v97xj" event={"ID":"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b","Type":"ContainerStarted","Data":"29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56"} Apr 16 20:42:11.296989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:11.296821 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7754d9dc6b-v97xj" event={"ID":"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b","Type":"ContainerStarted","Data":"4b89e31779d7dcb12e9a3626376e1856b55286df6a5c79284705cd135349e4f3"} Apr 16 20:42:11.314266 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:11.314225 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7754d9dc6b-v97xj" podStartSLOduration=1.314206884 podStartE2EDuration="1.314206884s" podCreationTimestamp="2026-04-16 20:42:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 20:42:11.312334844 +0000 UTC m=+261.487699109" watchObservedRunningTime="2026-04-16 20:42:11.314206884 +0000 UTC m=+261.489571149" Apr 16 20:42:20.612185 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:20.612148 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:20.612185 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:20.612194 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:20.616817 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:20.616793 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:21.329750 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:21.329720 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:42:21.381684 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:21.381621 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c7b887fd4-zx2xn"] Apr 16 20:42:46.405210 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.405149 2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-c7b887fd4-zx2xn" podUID="381b6539-2390-41f0-b3b1-50783e6e6649" containerName="console" containerID="cri-o://0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d" gracePeriod=15 Apr 16 20:42:46.639146 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.639126 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c7b887fd4-zx2xn_381b6539-2390-41f0-b3b1-50783e6e6649/console/0.log" Apr 16 20:42:46.639239 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.639186 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:42:46.820316 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.820291 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-oauth-serving-cert\") pod \"381b6539-2390-41f0-b3b1-50783e6e6649\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " Apr 16 20:42:46.820479 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.820328 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-trusted-ca-bundle\") pod \"381b6539-2390-41f0-b3b1-50783e6e6649\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " Apr 16 20:42:46.820479 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.820345 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-console-config\") pod \"381b6539-2390-41f0-b3b1-50783e6e6649\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " Apr 16 20:42:46.820479 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.820367 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-service-ca\") pod \"381b6539-2390-41f0-b3b1-50783e6e6649\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " Apr 16 20:42:46.820479 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.820434 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-serving-cert\") pod \"381b6539-2390-41f0-b3b1-50783e6e6649\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " Apr 16 20:42:46.820479 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.820460 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8m27\" (UniqueName: \"kubernetes.io/projected/381b6539-2390-41f0-b3b1-50783e6e6649-kube-api-access-t8m27\") pod \"381b6539-2390-41f0-b3b1-50783e6e6649\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " Apr 16 20:42:46.820770 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.820491 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-oauth-config\") pod \"381b6539-2390-41f0-b3b1-50783e6e6649\" (UID: \"381b6539-2390-41f0-b3b1-50783e6e6649\") " Apr 16 20:42:46.820770 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.820756 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "381b6539-2390-41f0-b3b1-50783e6e6649" (UID: "381b6539-2390-41f0-b3b1-50783e6e6649"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:42:46.820870 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.820826 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "381b6539-2390-41f0-b3b1-50783e6e6649" (UID: "381b6539-2390-41f0-b3b1-50783e6e6649"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:42:46.820870 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.820857 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-service-ca" (OuterVolumeSpecName: "service-ca") pod "381b6539-2390-41f0-b3b1-50783e6e6649" (UID: "381b6539-2390-41f0-b3b1-50783e6e6649"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:42:46.821092 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.821062 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-console-config" (OuterVolumeSpecName: "console-config") pod "381b6539-2390-41f0-b3b1-50783e6e6649" (UID: "381b6539-2390-41f0-b3b1-50783e6e6649"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:42:46.822820 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.822792 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "381b6539-2390-41f0-b3b1-50783e6e6649" (UID: "381b6539-2390-41f0-b3b1-50783e6e6649"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 20:42:46.822919 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.822886 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381b6539-2390-41f0-b3b1-50783e6e6649-kube-api-access-t8m27" (OuterVolumeSpecName: "kube-api-access-t8m27") pod "381b6539-2390-41f0-b3b1-50783e6e6649" (UID: "381b6539-2390-41f0-b3b1-50783e6e6649"). InnerVolumeSpecName "kube-api-access-t8m27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:42:46.822985 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.822939 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "381b6539-2390-41f0-b3b1-50783e6e6649" (UID: "381b6539-2390-41f0-b3b1-50783e6e6649"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 20:42:46.921676 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.921635 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-trusted-ca-bundle\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:42:46.921676 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.921674 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-console-config\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:42:46.921676 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.921684 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-service-ca\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:42:46.921877 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.921693 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-serving-cert\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:42:46.921877 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.921702 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8m27\" (UniqueName: \"kubernetes.io/projected/381b6539-2390-41f0-b3b1-50783e6e6649-kube-api-access-t8m27\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:42:46.921877 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.921712 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/381b6539-2390-41f0-b3b1-50783e6e6649-console-oauth-config\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:42:46.921877 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:46.921721 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/381b6539-2390-41f0-b3b1-50783e6e6649-oauth-serving-cert\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:42:47.398076 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:47.398049 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c7b887fd4-zx2xn_381b6539-2390-41f0-b3b1-50783e6e6649/console/0.log" Apr 16 20:42:47.398250 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:47.398086 2562 generic.go:358] "Generic (PLEG): container finished" podID="381b6539-2390-41f0-b3b1-50783e6e6649" containerID="0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d" exitCode=2 Apr 16 20:42:47.398250 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:47.398155 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7b887fd4-zx2xn" event={"ID":"381b6539-2390-41f0-b3b1-50783e6e6649","Type":"ContainerDied","Data":"0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d"} Apr 16 20:42:47.398250 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:47.398163 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7b887fd4-zx2xn" Apr 16 20:42:47.398250 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:47.398180 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7b887fd4-zx2xn" event={"ID":"381b6539-2390-41f0-b3b1-50783e6e6649","Type":"ContainerDied","Data":"4df49671682578c4cf114c42a54ad2924fbde81d171b7139ad179a5f4c60c116"} Apr 16 20:42:47.398250 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:47.398194 2562 scope.go:117] "RemoveContainer" containerID="0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d" Apr 16 20:42:47.406703 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:47.406428 2562 scope.go:117] "RemoveContainer" containerID="0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d" Apr 16 20:42:47.406703 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:42:47.406690 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d\": container with ID starting with 0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d not found: ID does not exist" containerID="0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d" Apr 16 20:42:47.406971 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:47.406715 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d"} err="failed to get container status \"0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d\": rpc error: code = NotFound desc = could not find container \"0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d\": container with ID starting with 0313d488423d7b2a4f5026655f84fc6d6d29320fe246e2f990a0e9dcd139902d not found: ID does not exist" Apr 16 20:42:47.418386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:47.418363 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c7b887fd4-zx2xn"] Apr 16 20:42:47.424329 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:47.424311 2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c7b887fd4-zx2xn"] Apr 16 20:42:48.500394 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:48.500358 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381b6539-2390-41f0-b3b1-50783e6e6649" path="/var/lib/kubelet/pods/381b6539-2390-41f0-b3b1-50783e6e6649/volumes" Apr 16 20:42:50.402505 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:50.402471 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:42:50.402955 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:50.402665 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:42:50.412304 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:42:50.412280 2562 kubelet.go:1628] "Image garbage collection succeeded" Apr 16 20:43:15.616903 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.616851 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-bf85fbb7d-wth7c"] Apr 16 20:43:15.619458 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.617268 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381b6539-2390-41f0-b3b1-50783e6e6649" containerName="console" Apr 16 20:43:15.619458 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.617290 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="381b6539-2390-41f0-b3b1-50783e6e6649" containerName="console" Apr 16 20:43:15.619458 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.617355 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="381b6539-2390-41f0-b3b1-50783e6e6649" containerName="console" Apr 16 20:43:15.620432 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.620411 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.630552 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.630528 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bf85fbb7d-wth7c"] Apr 16 20:43:15.715003 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.714979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-oauth-config\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.715157 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.715012 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-serving-cert\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.715157 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.715049 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x92jr\" (UniqueName: \"kubernetes.io/projected/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-kube-api-access-x92jr\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.715157 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.715095 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-config\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.715314 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.715160 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-oauth-serving-cert\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.715314 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.715195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-service-ca\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.715314 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.715266 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-trusted-ca-bundle\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.815524 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.815501 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-oauth-serving-cert\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.815682 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.815529 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-service-ca\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.815682 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.815545 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-trusted-ca-bundle\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.815682 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.815580 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-oauth-config\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.815682 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.815615 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-serving-cert\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.815682 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.815648 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x92jr\" (UniqueName: \"kubernetes.io/projected/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-kube-api-access-x92jr\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.815682 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.815678 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-config\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.816657 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.816591 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-service-ca\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.816772 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.816591 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-oauth-serving-cert\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.816835 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.816796 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-config\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.821063 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.817147 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-trusted-ca-bundle\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.821063 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.818644 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-oauth-config\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.821063 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.818903 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-serving-cert\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.824004 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.823985 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x92jr\" (UniqueName: \"kubernetes.io/projected/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-kube-api-access-x92jr\") pod \"console-bf85fbb7d-wth7c\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:15.930552 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:15.930504 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:16.048226 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:16.048198 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bf85fbb7d-wth7c"] Apr 16 20:43:16.051827 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:43:16.051792 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c57ea5_7dbc_400c_94fa_ddd7f94288a1.slice/crio-a5bf33b244e9957c7b024821bfabc5a4f60533a4cd028eefe3bbaf7324f2e1ea WatchSource:0}: Error finding container a5bf33b244e9957c7b024821bfabc5a4f60533a4cd028eefe3bbaf7324f2e1ea: Status 404 returned error can't find the container with id a5bf33b244e9957c7b024821bfabc5a4f60533a4cd028eefe3bbaf7324f2e1ea Apr 16 20:43:16.053554 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:16.053537 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 20:43:16.479398 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:16.479366 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf85fbb7d-wth7c" event={"ID":"78c57ea5-7dbc-400c-94fa-ddd7f94288a1","Type":"ContainerStarted","Data":"a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353"} Apr 16 20:43:16.479398 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:16.479402 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf85fbb7d-wth7c" event={"ID":"78c57ea5-7dbc-400c-94fa-ddd7f94288a1","Type":"ContainerStarted","Data":"a5bf33b244e9957c7b024821bfabc5a4f60533a4cd028eefe3bbaf7324f2e1ea"} Apr 16 20:43:16.496089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:16.496041 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-bf85fbb7d-wth7c" podStartSLOduration=1.496027297 podStartE2EDuration="1.496027297s" podCreationTimestamp="2026-04-16 20:43:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 20:43:16.494679003 +0000 UTC m=+326.670043269" watchObservedRunningTime="2026-04-16 20:43:16.496027297 +0000 UTC m=+326.671391563" Apr 16 20:43:25.931092 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:25.931062 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:25.931388 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:25.931103 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:25.935901 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:25.935878 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:26.514752 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:26.514729 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:43:26.558156 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:26.558128 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7754d9dc6b-v97xj"] Apr 16 20:43:51.588584 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.588488 2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7754d9dc6b-v97xj" podUID="987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" containerName="console" containerID="cri-o://29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56" gracePeriod=15 Apr 16 20:43:51.821707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.821687 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7754d9dc6b-v97xj_987fefec-8d26-4f7b-b9b2-e2cc90a63d2b/console/0.log" Apr 16 20:43:51.821817 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.821748 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:43:51.939554 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.939496 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-service-ca\") pod \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " Apr 16 20:43:51.939681 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.939559 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-config\") pod \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " Apr 16 20:43:51.939681 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.939620 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-oauth-serving-cert\") pod \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " Apr 16 20:43:51.939681 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.939643 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-oauth-config\") pod \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " Apr 16 20:43:51.939681 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.939661 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-serving-cert\") pod \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " Apr 16 20:43:51.939889 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.939684 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-trusted-ca-bundle\") pod \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " Apr 16 20:43:51.939889 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.939728 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nr8q\" (UniqueName: \"kubernetes.io/projected/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-kube-api-access-7nr8q\") pod \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\" (UID: \"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b\") " Apr 16 20:43:51.939989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.939889 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-config" (OuterVolumeSpecName: "console-config") pod "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" (UID: "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:43:51.939989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.939878 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-service-ca" (OuterVolumeSpecName: "service-ca") pod "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" (UID: "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:43:51.940215 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.940183 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" (UID: "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:43:51.940330 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.940255 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" (UID: "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:43:51.941855 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.941830 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" (UID: "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 20:43:51.941950 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.941856 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" (UID: "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 20:43:51.941950 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:51.941899 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-kube-api-access-7nr8q" (OuterVolumeSpecName: "kube-api-access-7nr8q") pod "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" (UID: "987fefec-8d26-4f7b-b9b2-e2cc90a63d2b"). InnerVolumeSpecName "kube-api-access-7nr8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:43:52.040493 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.040467 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-serving-cert\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:43:52.040493 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.040492 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-trusted-ca-bundle\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:43:52.040668 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.040506 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nr8q\" (UniqueName: \"kubernetes.io/projected/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-kube-api-access-7nr8q\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:43:52.040668 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.040520 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-service-ca\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:43:52.040668 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.040535 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-config\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:43:52.040668 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.040546 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-oauth-serving-cert\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:43:52.040668 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.040560 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b-console-oauth-config\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:43:52.570639 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.570622 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7754d9dc6b-v97xj_987fefec-8d26-4f7b-b9b2-e2cc90a63d2b/console/0.log" Apr 16 20:43:52.570736 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.570656 2562 generic.go:358] "Generic (PLEG): container finished" podID="987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" containerID="29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56" exitCode=2 Apr 16 20:43:52.570736 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.570707 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7754d9dc6b-v97xj" event={"ID":"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b","Type":"ContainerDied","Data":"29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56"} Apr 16 20:43:52.570736 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.570714 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7754d9dc6b-v97xj" Apr 16 20:43:52.570736 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.570725 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7754d9dc6b-v97xj" event={"ID":"987fefec-8d26-4f7b-b9b2-e2cc90a63d2b","Type":"ContainerDied","Data":"4b89e31779d7dcb12e9a3626376e1856b55286df6a5c79284705cd135349e4f3"} Apr 16 20:43:52.570863 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.570740 2562 scope.go:117] "RemoveContainer" containerID="29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56" Apr 16 20:43:52.577912 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.577896 2562 scope.go:117] "RemoveContainer" containerID="29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56" Apr 16 20:43:52.578130 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:43:52.578112 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56\": container with ID starting with 29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56 not found: ID does not exist" containerID="29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56" Apr 16 20:43:52.578179 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.578136 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56"} err="failed to get container status \"29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56\": rpc error: code = NotFound desc = could not find container \"29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56\": container with ID starting with 29661e8d0e632611d9bea14c822c200e1fc880c5e0bc6ac69557e7924d995f56 not found: ID does not exist" Apr 16 20:43:52.588290 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.588270 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7754d9dc6b-v97xj"] Apr 16 20:43:52.592693 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:52.592672 2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7754d9dc6b-v97xj"] Apr 16 20:43:54.501428 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:43:54.501396 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" path="/var/lib/kubelet/pods/987fefec-8d26-4f7b-b9b2-e2cc90a63d2b/volumes" Apr 16 20:44:17.909494 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:17.909457 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf"] Apr 16 20:44:17.909874 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:17.909740 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" containerName="console" Apr 16 20:44:17.909874 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:17.909752 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" containerName="console" Apr 16 20:44:17.909874 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:17.909809 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="987fefec-8d26-4f7b-b9b2-e2cc90a63d2b" containerName="console" Apr 16 20:44:17.912804 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:17.912785 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:17.915280 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:17.915257 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Apr 16 20:44:17.915389 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:17.915260 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Apr 16 20:44:17.916352 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:17.916333 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7g5mm\"" Apr 16 20:44:17.921831 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:17.921808 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf"] Apr 16 20:44:18.011657 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.011634 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-bundle\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:18.011748 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.011666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-util\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:18.011748 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.011733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rl6m\" (UniqueName: \"kubernetes.io/projected/27675436-48af-4458-aa15-91ed37fa949b-kube-api-access-2rl6m\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:18.112673 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.112650 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rl6m\" (UniqueName: \"kubernetes.io/projected/27675436-48af-4458-aa15-91ed37fa949b-kube-api-access-2rl6m\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:18.112774 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.112702 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-bundle\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:18.112774 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.112739 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-util\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:18.113059 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.113039 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-bundle\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:18.113136 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.113080 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-util\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:18.120785 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.120758 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rl6m\" (UniqueName: \"kubernetes.io/projected/27675436-48af-4458-aa15-91ed37fa949b-kube-api-access-2rl6m\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:18.221937 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.221882 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:18.336878 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.336798 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf"] Apr 16 20:44:18.339674 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:44:18.339642 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27675436_48af_4458_aa15_91ed37fa949b.slice/crio-0cf0f5b8e5c655a428d44170978048ff5ed12365a698178d49f28ead05f497ff WatchSource:0}: Error finding container 0cf0f5b8e5c655a428d44170978048ff5ed12365a698178d49f28ead05f497ff: Status 404 returned error can't find the container with id 0cf0f5b8e5c655a428d44170978048ff5ed12365a698178d49f28ead05f497ff Apr 16 20:44:18.632855 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:18.632826 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" event={"ID":"27675436-48af-4458-aa15-91ed37fa949b","Type":"ContainerStarted","Data":"0cf0f5b8e5c655a428d44170978048ff5ed12365a698178d49f28ead05f497ff"} Apr 16 20:44:25.655738 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.655670 2562 generic.go:358] "Generic (PLEG): container finished" podID="27675436-48af-4458-aa15-91ed37fa949b" containerID="163c06ec35180235cda1ce43e9ccc2402b8cb20e64551249084113cdd1df455c" exitCode=0 Apr 16 20:44:25.655738 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.655721 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" event={"ID":"27675436-48af-4458-aa15-91ed37fa949b","Type":"ContainerDied","Data":"163c06ec35180235cda1ce43e9ccc2402b8cb20e64551249084113cdd1df455c"} Apr 16 20:44:25.948963 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.948898 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth"] Apr 16 20:44:25.951951 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.951928 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd"] Apr 16 20:44:25.952101 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.952081 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:25.954684 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.954655 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Apr 16 20:44:25.954788 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.954659 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Apr 16 20:44:25.954788 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.954659 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Apr 16 20:44:25.955023 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.955002 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Apr 16 20:44:25.955140 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.955123 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:25.957295 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.957271 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Apr 16 20:44:25.957379 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.957368 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Apr 16 20:44:25.957479 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.957455 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Apr 16 20:44:25.957644 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.957626 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Apr 16 20:44:25.960498 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.960474 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth"] Apr 16 20:44:25.963049 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:25.963023 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd"] Apr 16 20:44:26.075896 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.075871 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.076011 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.075899 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.076011 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.075930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-hub\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.076082 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.076017 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4620bade-e035-4e39-b675-f249c75b0206-tmp\") pod \"klusterlet-addon-workmgr-7bf887697-m8mth\" (UID: \"4620bade-e035-4e39-b675-f249c75b0206\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:26.076082 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.076043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4620bade-e035-4e39-b675-f249c75b0206-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf887697-m8mth\" (UID: \"4620bade-e035-4e39-b675-f249c75b0206\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:26.076082 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.076070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-ca\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.076175 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.076110 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt944\" (UniqueName: \"kubernetes.io/projected/4620bade-e035-4e39-b675-f249c75b0206-kube-api-access-dt944\") pod \"klusterlet-addon-workmgr-7bf887697-m8mth\" (UID: \"4620bade-e035-4e39-b675-f249c75b0206\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:26.076175 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.076165 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.076243 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.076187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg79l\" (UniqueName: \"kubernetes.io/projected/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-kube-api-access-gg79l\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.176570 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.176541 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.176688 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.176576 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.176688 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.176625 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-hub\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.176688 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.176654 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4620bade-e035-4e39-b675-f249c75b0206-tmp\") pod \"klusterlet-addon-workmgr-7bf887697-m8mth\" (UID: \"4620bade-e035-4e39-b675-f249c75b0206\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:26.176688 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.176673 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4620bade-e035-4e39-b675-f249c75b0206-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf887697-m8mth\" (UID: \"4620bade-e035-4e39-b675-f249c75b0206\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:26.176895 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.176701 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-ca\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.176895 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.176741 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dt944\" (UniqueName: \"kubernetes.io/projected/4620bade-e035-4e39-b675-f249c75b0206-kube-api-access-dt944\") pod \"klusterlet-addon-workmgr-7bf887697-m8mth\" (UID: \"4620bade-e035-4e39-b675-f249c75b0206\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:26.176895 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.176789 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.176895 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.176826 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gg79l\" (UniqueName: \"kubernetes.io/projected/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-kube-api-access-gg79l\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.177454 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.177409 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4620bade-e035-4e39-b675-f249c75b0206-tmp\") pod \"klusterlet-addon-workmgr-7bf887697-m8mth\" (UID: \"4620bade-e035-4e39-b675-f249c75b0206\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:26.177454 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.177438 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.179253 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.179230 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.179660 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.179637 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-ca\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.179731 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.179675 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-hub\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.179731 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.179709 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.179834 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.179765 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4620bade-e035-4e39-b675-f249c75b0206-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf887697-m8mth\" (UID: \"4620bade-e035-4e39-b675-f249c75b0206\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:26.184506 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.184484 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt944\" (UniqueName: \"kubernetes.io/projected/4620bade-e035-4e39-b675-f249c75b0206-kube-api-access-dt944\") pod \"klusterlet-addon-workmgr-7bf887697-m8mth\" (UID: \"4620bade-e035-4e39-b675-f249c75b0206\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:26.184613 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.184530 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg79l\" (UniqueName: \"kubernetes.io/projected/9759ef03-d5f3-491c-b11c-824f7fa0a4dd-kube-api-access-gg79l\") pod \"cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd\" (UID: \"9759ef03-d5f3-491c-b11c-824f7fa0a4dd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.263791 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.263739 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:26.277631 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.277596 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" Apr 16 20:44:26.388817 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.388787 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth"] Apr 16 20:44:26.391831 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:44:26.391802 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4620bade_e035_4e39_b675_f249c75b0206.slice/crio-89e965eebc0e68c940fa96e9beb2919af05a3a39eb0bb0687cb7027fc2231fd5 WatchSource:0}: Error finding container 89e965eebc0e68c940fa96e9beb2919af05a3a39eb0bb0687cb7027fc2231fd5: Status 404 returned error can't find the container with id 89e965eebc0e68c940fa96e9beb2919af05a3a39eb0bb0687cb7027fc2231fd5 Apr 16 20:44:26.411676 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.411649 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd"] Apr 16 20:44:26.414665 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:44:26.414641 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9759ef03_d5f3_491c_b11c_824f7fa0a4dd.slice/crio-94f607b1dc24a354b0f5ff5525786638a4cae24e6bf2d8a6b151bfdd9e7c969b WatchSource:0}: Error finding container 94f607b1dc24a354b0f5ff5525786638a4cae24e6bf2d8a6b151bfdd9e7c969b: Status 404 returned error can't find the container with id 94f607b1dc24a354b0f5ff5525786638a4cae24e6bf2d8a6b151bfdd9e7c969b Apr 16 20:44:26.659946 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.659914 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" event={"ID":"9759ef03-d5f3-491c-b11c-824f7fa0a4dd","Type":"ContainerStarted","Data":"94f607b1dc24a354b0f5ff5525786638a4cae24e6bf2d8a6b151bfdd9e7c969b"} Apr 16 20:44:26.660899 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:26.660874 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" event={"ID":"4620bade-e035-4e39-b675-f249c75b0206","Type":"ContainerStarted","Data":"89e965eebc0e68c940fa96e9beb2919af05a3a39eb0bb0687cb7027fc2231fd5"} Apr 16 20:44:31.678074 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:31.678040 2562 generic.go:358] "Generic (PLEG): container finished" podID="27675436-48af-4458-aa15-91ed37fa949b" containerID="52d7ef86033ee65b1a695e4b4784ed132a659ab5820c0f3c115f90e333fe53f0" exitCode=0 Apr 16 20:44:31.678426 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:31.678128 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" event={"ID":"27675436-48af-4458-aa15-91ed37fa949b","Type":"ContainerDied","Data":"52d7ef86033ee65b1a695e4b4784ed132a659ab5820c0f3c115f90e333fe53f0"} Apr 16 20:44:31.679502 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:31.679451 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" event={"ID":"4620bade-e035-4e39-b675-f249c75b0206","Type":"ContainerStarted","Data":"7e160386d5a66667785619cd48db33ec8677a87fc05aa869218b4d029058ab14"} Apr 16 20:44:31.679622 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:31.679561 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:31.680859 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:31.680844 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" event={"ID":"9759ef03-d5f3-491c-b11c-824f7fa0a4dd","Type":"ContainerStarted","Data":"86147c806d7de265b67ce48258e5365dedfae130586fa8b1e6fb5a3232447fbe"} Apr 16 20:44:31.681643 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:31.681623 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" Apr 16 20:44:31.709052 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:31.708976 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf887697-m8mth" podStartSLOduration=1.642128596 podStartE2EDuration="6.708959624s" podCreationTimestamp="2026-04-16 20:44:25 +0000 UTC" firstStartedPulling="2026-04-16 20:44:26.393934792 +0000 UTC m=+396.569299042" lastFinishedPulling="2026-04-16 20:44:31.460765813 +0000 UTC m=+401.636130070" observedRunningTime="2026-04-16 20:44:31.708341844 +0000 UTC m=+401.883706111" watchObservedRunningTime="2026-04-16 20:44:31.708959624 +0000 UTC m=+401.884323894" Apr 16 20:44:33.689177 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:33.689142 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" event={"ID":"9759ef03-d5f3-491c-b11c-824f7fa0a4dd","Type":"ContainerStarted","Data":"b5e4f06c22c8161bce12f71898c46d22b68a1e3e41706deb859bb52e86e39e0e"} Apr 16 20:44:33.689515 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:33.689183 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" event={"ID":"9759ef03-d5f3-491c-b11c-824f7fa0a4dd","Type":"ContainerStarted","Data":"cf8b51cf21ac0c591f7ade249b32e59b07e3ba20ab95564c4edfb7fc603a1812"} Apr 16 20:44:33.707923 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:33.707875 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c8c7b5ddd-2frfd" podStartSLOduration=1.6362256080000002 podStartE2EDuration="8.707862578s" podCreationTimestamp="2026-04-16 20:44:25 +0000 UTC" firstStartedPulling="2026-04-16 20:44:26.416743633 +0000 UTC m=+396.592107880" lastFinishedPulling="2026-04-16 20:44:33.488380603 +0000 UTC m=+403.663744850" observedRunningTime="2026-04-16 20:44:33.706090061 +0000 UTC m=+403.881454364" watchObservedRunningTime="2026-04-16 20:44:33.707862578 +0000 UTC m=+403.883226844" Apr 16 20:44:39.709704 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:39.709679 2562 generic.go:358] "Generic (PLEG): container finished" podID="27675436-48af-4458-aa15-91ed37fa949b" containerID="68584e8029c2141ce928c7bf7618d6e15da04bd2441bc921eaad59488dd34db3" exitCode=0 Apr 16 20:44:39.709990 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:39.709725 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" event={"ID":"27675436-48af-4458-aa15-91ed37fa949b","Type":"ContainerDied","Data":"68584e8029c2141ce928c7bf7618d6e15da04bd2441bc921eaad59488dd34db3"} Apr 16 20:44:40.825905 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:40.825881 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:40.998491 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:40.998436 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-util\") pod \"27675436-48af-4458-aa15-91ed37fa949b\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " Apr 16 20:44:40.998623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:40.998528 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-bundle\") pod \"27675436-48af-4458-aa15-91ed37fa949b\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " Apr 16 20:44:40.998623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:40.998561 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rl6m\" (UniqueName: \"kubernetes.io/projected/27675436-48af-4458-aa15-91ed37fa949b-kube-api-access-2rl6m\") pod \"27675436-48af-4458-aa15-91ed37fa949b\" (UID: \"27675436-48af-4458-aa15-91ed37fa949b\") " Apr 16 20:44:40.999120 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:40.999094 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-bundle" (OuterVolumeSpecName: "bundle") pod "27675436-48af-4458-aa15-91ed37fa949b" (UID: "27675436-48af-4458-aa15-91ed37fa949b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:44:41.000644 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:41.000599 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27675436-48af-4458-aa15-91ed37fa949b-kube-api-access-2rl6m" (OuterVolumeSpecName: "kube-api-access-2rl6m") pod "27675436-48af-4458-aa15-91ed37fa949b" (UID: "27675436-48af-4458-aa15-91ed37fa949b"). InnerVolumeSpecName "kube-api-access-2rl6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:44:41.002399 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:41.002375 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-util" (OuterVolumeSpecName: "util") pod "27675436-48af-4458-aa15-91ed37fa949b" (UID: "27675436-48af-4458-aa15-91ed37fa949b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:44:41.099897 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:41.099877 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-util\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:44:41.099989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:41.099900 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27675436-48af-4458-aa15-91ed37fa949b-bundle\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:44:41.099989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:41.099914 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rl6m\" (UniqueName: \"kubernetes.io/projected/27675436-48af-4458-aa15-91ed37fa949b-kube-api-access-2rl6m\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:44:41.717044 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:41.717011 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" event={"ID":"27675436-48af-4458-aa15-91ed37fa949b","Type":"ContainerDied","Data":"0cf0f5b8e5c655a428d44170978048ff5ed12365a698178d49f28ead05f497ff"} Apr 16 20:44:41.717044 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:41.717028 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5bplbf" Apr 16 20:44:41.717044 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:44:41.717045 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cf0f5b8e5c655a428d44170978048ff5ed12365a698178d49f28ead05f497ff" Apr 16 20:45:20.523331 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.523298 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s"] Apr 16 20:45:20.523875 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.523598 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27675436-48af-4458-aa15-91ed37fa949b" containerName="extract" Apr 16 20:45:20.523875 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.523627 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27675436-48af-4458-aa15-91ed37fa949b" containerName="extract" Apr 16 20:45:20.523875 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.523641 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27675436-48af-4458-aa15-91ed37fa949b" containerName="pull" Apr 16 20:45:20.523875 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.523649 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27675436-48af-4458-aa15-91ed37fa949b" containerName="pull" Apr 16 20:45:20.523875 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.523659 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27675436-48af-4458-aa15-91ed37fa949b" containerName="util" Apr 16 20:45:20.523875 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.523667 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27675436-48af-4458-aa15-91ed37fa949b" containerName="util" Apr 16 20:45:20.523875 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.523738 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="27675436-48af-4458-aa15-91ed37fa949b" containerName="extract" Apr 16 20:45:20.527233 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.527212 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.529866 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.529841 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Apr 16 20:45:20.530950 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.530933 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Apr 16 20:45:20.531012 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.530933 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7g5mm\"" Apr 16 20:45:20.535691 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.535672 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s"] Apr 16 20:45:20.553062 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.553041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-util\") pod \"3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.553166 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.553074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xjvg\" (UniqueName: \"kubernetes.io/projected/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-kube-api-access-5xjvg\") pod \"3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.553166 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.553101 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-bundle\") pod \"3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.653517 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.653492 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-util\") pod \"3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.653593 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.653539 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5xjvg\" (UniqueName: \"kubernetes.io/projected/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-kube-api-access-5xjvg\") pod \"3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.653593 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.653566 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-bundle\") pod \"3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.653800 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.653785 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-util\") pod \"3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.653888 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.653872 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-bundle\") pod \"3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.661424 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.661406 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xjvg\" (UniqueName: \"kubernetes.io/projected/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-kube-api-access-5xjvg\") pod \"3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.837008 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.836981 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:20.957491 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:20.957449 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s"] Apr 16 20:45:20.965588 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:45:20.965552 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfdda20e_80bf_48cf_a0e5_5bb398f26e76.slice/crio-d983800efc4e31f2c9f6b72971a5d037553c3d7a242e85745c7e8aa59b5ec2e0 WatchSource:0}: Error finding container d983800efc4e31f2c9f6b72971a5d037553c3d7a242e85745c7e8aa59b5ec2e0: Status 404 returned error can't find the container with id d983800efc4e31f2c9f6b72971a5d037553c3d7a242e85745c7e8aa59b5ec2e0 Apr 16 20:45:21.830224 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:21.830188 2562 generic.go:358] "Generic (PLEG): container finished" podID="cfdda20e-80bf-48cf-a0e5-5bb398f26e76" containerID="26f2dd0f180866ebde48fdf09813ee744e367e536db4cd5ddb3b08ae1f5caa25" exitCode=0 Apr 16 20:45:21.830561 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:21.830277 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" event={"ID":"cfdda20e-80bf-48cf-a0e5-5bb398f26e76","Type":"ContainerDied","Data":"26f2dd0f180866ebde48fdf09813ee744e367e536db4cd5ddb3b08ae1f5caa25"} Apr 16 20:45:21.830561 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:21.830310 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" event={"ID":"cfdda20e-80bf-48cf-a0e5-5bb398f26e76","Type":"ContainerStarted","Data":"d983800efc4e31f2c9f6b72971a5d037553c3d7a242e85745c7e8aa59b5ec2e0"} Apr 16 20:45:22.835425 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:22.835394 2562 generic.go:358] "Generic (PLEG): container finished" podID="cfdda20e-80bf-48cf-a0e5-5bb398f26e76" containerID="8c9c4f86104f2b36bc2aa4732424d0bf41ff02baaac48386cf2f6cc0d766f25f" exitCode=0 Apr 16 20:45:22.835746 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:22.835479 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" event={"ID":"cfdda20e-80bf-48cf-a0e5-5bb398f26e76","Type":"ContainerDied","Data":"8c9c4f86104f2b36bc2aa4732424d0bf41ff02baaac48386cf2f6cc0d766f25f"} Apr 16 20:45:23.839822 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:23.839789 2562 generic.go:358] "Generic (PLEG): container finished" podID="cfdda20e-80bf-48cf-a0e5-5bb398f26e76" containerID="1bf8b80c2ecf719e2115adbc82302cd54a1d6d3cdf8f929586bc3ee945a92744" exitCode=0 Apr 16 20:45:23.840157 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:23.839882 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" event={"ID":"cfdda20e-80bf-48cf-a0e5-5bb398f26e76","Type":"ContainerDied","Data":"1bf8b80c2ecf719e2115adbc82302cd54a1d6d3cdf8f929586bc3ee945a92744"} Apr 16 20:45:24.752141 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.752108 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp"] Apr 16 20:45:24.755095 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.755079 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.757708 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.757683 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Apr 16 20:45:24.757825 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.757706 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Apr 16 20:45:24.757825 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.757796 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Apr 16 20:45:24.759020 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.759000 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Apr 16 20:45:24.759107 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.759000 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-d5gf4\"" Apr 16 20:45:24.759107 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.759042 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Apr 16 20:45:24.766416 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.766397 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp"] Apr 16 20:45:24.784263 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.784235 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6dad5e38-aa96-4695-9fb2-6f876421925f-manager-config\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.784337 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.784268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dad5e38-aa96-4695-9fb2-6f876421925f-cert\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.784337 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.784304 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlc4\" (UniqueName: \"kubernetes.io/projected/6dad5e38-aa96-4695-9fb2-6f876421925f-kube-api-access-lmlc4\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.784417 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.784371 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6dad5e38-aa96-4695-9fb2-6f876421925f-metrics-cert\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.885345 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.885303 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6dad5e38-aa96-4695-9fb2-6f876421925f-metrics-cert\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.885749 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.885370 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6dad5e38-aa96-4695-9fb2-6f876421925f-manager-config\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.885749 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.885399 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dad5e38-aa96-4695-9fb2-6f876421925f-cert\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.885749 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.885436 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlc4\" (UniqueName: \"kubernetes.io/projected/6dad5e38-aa96-4695-9fb2-6f876421925f-kube-api-access-lmlc4\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.886251 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.886229 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6dad5e38-aa96-4695-9fb2-6f876421925f-manager-config\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.888258 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.888234 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6dad5e38-aa96-4695-9fb2-6f876421925f-metrics-cert\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.888434 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.888411 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dad5e38-aa96-4695-9fb2-6f876421925f-cert\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.893952 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.893930 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlc4\" (UniqueName: \"kubernetes.io/projected/6dad5e38-aa96-4695-9fb2-6f876421925f-kube-api-access-lmlc4\") pod \"lws-controller-manager-5494fc4578-7c4xp\" (UID: \"6dad5e38-aa96-4695-9fb2-6f876421925f\") " pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:24.958703 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.958684 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:24.986218 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.986199 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-util\") pod \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " Apr 16 20:45:24.986335 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.986228 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xjvg\" (UniqueName: \"kubernetes.io/projected/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-kube-api-access-5xjvg\") pod \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " Apr 16 20:45:24.986335 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.986260 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-bundle\") pod \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\" (UID: \"cfdda20e-80bf-48cf-a0e5-5bb398f26e76\") " Apr 16 20:45:24.986955 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.986933 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-bundle" (OuterVolumeSpecName: "bundle") pod "cfdda20e-80bf-48cf-a0e5-5bb398f26e76" (UID: "cfdda20e-80bf-48cf-a0e5-5bb398f26e76"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:45:24.988081 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.988048 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-kube-api-access-5xjvg" (OuterVolumeSpecName: "kube-api-access-5xjvg") pod "cfdda20e-80bf-48cf-a0e5-5bb398f26e76" (UID: "cfdda20e-80bf-48cf-a0e5-5bb398f26e76"). InnerVolumeSpecName "kube-api-access-5xjvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:45:24.994723 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:24.994699 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-util" (OuterVolumeSpecName: "util") pod "cfdda20e-80bf-48cf-a0e5-5bb398f26e76" (UID: "cfdda20e-80bf-48cf-a0e5-5bb398f26e76"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:45:25.064368 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:25.064348 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:25.087092 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:25.087072 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-util\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:45:25.087092 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:25.087092 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xjvg\" (UniqueName: \"kubernetes.io/projected/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-kube-api-access-5xjvg\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:45:25.087222 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:25.087102 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfdda20e-80bf-48cf-a0e5-5bb398f26e76-bundle\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:45:25.176088 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:25.176059 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp"] Apr 16 20:45:25.178738 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:45:25.178709 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dad5e38_aa96_4695_9fb2_6f876421925f.slice/crio-4c95fe09cf1893aeebbac12ad70961299b15a6da026972fa96dbf8af498a3c81 WatchSource:0}: Error finding container 4c95fe09cf1893aeebbac12ad70961299b15a6da026972fa96dbf8af498a3c81: Status 404 returned error can't find the container with id 4c95fe09cf1893aeebbac12ad70961299b15a6da026972fa96dbf8af498a3c81 Apr 16 20:45:25.850962 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:25.850929 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" event={"ID":"cfdda20e-80bf-48cf-a0e5-5bb398f26e76","Type":"ContainerDied","Data":"d983800efc4e31f2c9f6b72971a5d037553c3d7a242e85745c7e8aa59b5ec2e0"} Apr 16 20:45:25.851150 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:25.850968 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d983800efc4e31f2c9f6b72971a5d037553c3d7a242e85745c7e8aa59b5ec2e0" Apr 16 20:45:25.851150 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:25.850967 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/3081035efbc3f76b8ae1b663abbd2a6bc5b4896fe94818011b247755c5w5c5s" Apr 16 20:45:25.852044 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:25.852023 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" event={"ID":"6dad5e38-aa96-4695-9fb2-6f876421925f","Type":"ContainerStarted","Data":"4c95fe09cf1893aeebbac12ad70961299b15a6da026972fa96dbf8af498a3c81"} Apr 16 20:45:28.861007 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:28.860974 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" event={"ID":"6dad5e38-aa96-4695-9fb2-6f876421925f","Type":"ContainerStarted","Data":"9290e65d759cf489f31f6deee1b9b1ac41e1a4253d89b5101784b26385a7f575"} Apr 16 20:45:28.861353 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:28.861118 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:28.883772 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:28.883726 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" podStartSLOduration=1.740721282 podStartE2EDuration="4.883713342s" podCreationTimestamp="2026-04-16 20:45:24 +0000 UTC" firstStartedPulling="2026-04-16 20:45:25.18053497 +0000 UTC m=+455.355899214" lastFinishedPulling="2026-04-16 20:45:28.32352703 +0000 UTC m=+458.498891274" observedRunningTime="2026-04-16 20:45:28.882174997 +0000 UTC m=+459.057539263" watchObservedRunningTime="2026-04-16 20:45:28.883713342 +0000 UTC m=+459.059077650" Apr 16 20:45:31.318224 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.318194 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx"] Apr 16 20:45:31.318573 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.318445 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdda20e-80bf-48cf-a0e5-5bb398f26e76" containerName="extract" Apr 16 20:45:31.318573 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.318455 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdda20e-80bf-48cf-a0e5-5bb398f26e76" containerName="extract" Apr 16 20:45:31.318573 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.318467 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdda20e-80bf-48cf-a0e5-5bb398f26e76" containerName="util" Apr 16 20:45:31.318573 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.318473 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdda20e-80bf-48cf-a0e5-5bb398f26e76" containerName="util" Apr 16 20:45:31.318573 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.318487 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdda20e-80bf-48cf-a0e5-5bb398f26e76" containerName="pull" Apr 16 20:45:31.318573 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.318492 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdda20e-80bf-48cf-a0e5-5bb398f26e76" containerName="pull" Apr 16 20:45:31.318573 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.318536 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="cfdda20e-80bf-48cf-a0e5-5bb398f26e76" containerName="extract" Apr 16 20:45:31.321514 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.321495 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.324000 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.323978 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7g5mm\"" Apr 16 20:45:31.324092 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.324001 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Apr 16 20:45:31.325113 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.325096 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Apr 16 20:45:31.335882 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.335463 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx"] Apr 16 20:45:31.432660 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.432631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-util\") pod \"f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.432767 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.432667 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvmq9\" (UniqueName: \"kubernetes.io/projected/09d50452-2c35-4ae2-931d-b7bbae2cedd8-kube-api-access-mvmq9\") pod \"f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.432767 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.432731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-bundle\") pod \"f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.533755 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.533727 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-util\") pod \"f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.533854 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.533759 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mvmq9\" (UniqueName: \"kubernetes.io/projected/09d50452-2c35-4ae2-931d-b7bbae2cedd8-kube-api-access-mvmq9\") pod \"f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.533854 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.533794 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-bundle\") pod \"f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.534148 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.534127 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-util\") pod \"f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.534148 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.534141 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-bundle\") pod \"f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.553223 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.553195 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvmq9\" (UniqueName: \"kubernetes.io/projected/09d50452-2c35-4ae2-931d-b7bbae2cedd8-kube-api-access-mvmq9\") pod \"f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.631282 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.631232 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:31.744173 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.744150 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx"] Apr 16 20:45:31.745946 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:45:31.745912 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d50452_2c35_4ae2_931d_b7bbae2cedd8.slice/crio-606500274b3497f7632fc3085ee0b864c0050e0455f17f96933a679a6c096b14 WatchSource:0}: Error finding container 606500274b3497f7632fc3085ee0b864c0050e0455f17f96933a679a6c096b14: Status 404 returned error can't find the container with id 606500274b3497f7632fc3085ee0b864c0050e0455f17f96933a679a6c096b14 Apr 16 20:45:31.869592 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.869567 2562 generic.go:358] "Generic (PLEG): container finished" podID="09d50452-2c35-4ae2-931d-b7bbae2cedd8" containerID="fef4f2e685b5950126e1af2b774559cc9a81d98391899c389259ff6b6ffb4c58" exitCode=0 Apr 16 20:45:31.869743 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.869624 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" event={"ID":"09d50452-2c35-4ae2-931d-b7bbae2cedd8","Type":"ContainerDied","Data":"fef4f2e685b5950126e1af2b774559cc9a81d98391899c389259ff6b6ffb4c58"} Apr 16 20:45:31.869743 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:31.869651 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" event={"ID":"09d50452-2c35-4ae2-931d-b7bbae2cedd8","Type":"ContainerStarted","Data":"606500274b3497f7632fc3085ee0b864c0050e0455f17f96933a679a6c096b14"} Apr 16 20:45:32.781934 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.781856 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln"] Apr 16 20:45:32.784872 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.784855 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:32.790772 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.790742 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Apr 16 20:45:32.792087 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.792069 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Apr 16 20:45:32.792568 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.792522 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Apr 16 20:45:32.792712 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.792696 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Apr 16 20:45:32.792868 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.792847 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lt7rc\"" Apr 16 20:45:32.815366 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.815335 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln"] Apr 16 20:45:32.844920 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.844896 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vqw\" (UniqueName: \"kubernetes.io/projected/edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc-kube-api-access-z6vqw\") pod \"opendatahub-operator-controller-manager-7cd8df7dd5-t8gln\" (UID: \"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:32.845008 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.844933 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cd8df7dd5-t8gln\" (UID: \"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:32.845008 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.844956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cd8df7dd5-t8gln\" (UID: \"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:32.873469 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.873441 2562 generic.go:358] "Generic (PLEG): container finished" podID="09d50452-2c35-4ae2-931d-b7bbae2cedd8" containerID="7d7801e5841174c8f2f99308a3fbaea0c5a2f45313ec1bc7b9f3db7bd518d011" exitCode=0 Apr 16 20:45:32.873558 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.873517 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" event={"ID":"09d50452-2c35-4ae2-931d-b7bbae2cedd8","Type":"ContainerDied","Data":"7d7801e5841174c8f2f99308a3fbaea0c5a2f45313ec1bc7b9f3db7bd518d011"} Apr 16 20:45:32.945902 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.945873 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cd8df7dd5-t8gln\" (UID: \"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:32.946021 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.945923 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cd8df7dd5-t8gln\" (UID: \"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:32.946069 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.946024 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vqw\" (UniqueName: \"kubernetes.io/projected/edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc-kube-api-access-z6vqw\") pod \"opendatahub-operator-controller-manager-7cd8df7dd5-t8gln\" (UID: \"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:32.948377 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.948354 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cd8df7dd5-t8gln\" (UID: \"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:32.948638 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.948619 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cd8df7dd5-t8gln\" (UID: \"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:32.956875 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:32.956833 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vqw\" (UniqueName: \"kubernetes.io/projected/edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc-kube-api-access-z6vqw\") pod \"opendatahub-operator-controller-manager-7cd8df7dd5-t8gln\" (UID: \"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:33.094800 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:33.094775 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:33.216007 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:33.215984 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln"] Apr 16 20:45:33.219224 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:45:33.219197 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf62ef4_faa3_4aec_82a5_54ddc0e8bcfc.slice/crio-5d9965dd39c09c9e42b171794a985da3e29b245408d63e7cbbebf6f062eccc1d WatchSource:0}: Error finding container 5d9965dd39c09c9e42b171794a985da3e29b245408d63e7cbbebf6f062eccc1d: Status 404 returned error can't find the container with id 5d9965dd39c09c9e42b171794a985da3e29b245408d63e7cbbebf6f062eccc1d Apr 16 20:45:33.882217 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:33.882135 2562 generic.go:358] "Generic (PLEG): container finished" podID="09d50452-2c35-4ae2-931d-b7bbae2cedd8" containerID="b1d7015ae2c6c25fb564ee070fe7f3ad8798e7317947a715833a44cc4aebc8a0" exitCode=0 Apr 16 20:45:33.882655 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:33.882227 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" event={"ID":"09d50452-2c35-4ae2-931d-b7bbae2cedd8","Type":"ContainerDied","Data":"b1d7015ae2c6c25fb564ee070fe7f3ad8798e7317947a715833a44cc4aebc8a0"} Apr 16 20:45:33.883680 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:33.883643 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" event={"ID":"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc","Type":"ContainerStarted","Data":"5d9965dd39c09c9e42b171794a985da3e29b245408d63e7cbbebf6f062eccc1d"} Apr 16 20:45:35.108852 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.108825 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:35.162898 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.162819 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-bundle\") pod \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " Apr 16 20:45:35.162898 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.162890 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-util\") pod \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " Apr 16 20:45:35.163106 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.162930 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvmq9\" (UniqueName: \"kubernetes.io/projected/09d50452-2c35-4ae2-931d-b7bbae2cedd8-kube-api-access-mvmq9\") pod \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\" (UID: \"09d50452-2c35-4ae2-931d-b7bbae2cedd8\") " Apr 16 20:45:35.163707 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.163670 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-bundle" (OuterVolumeSpecName: "bundle") pod "09d50452-2c35-4ae2-931d-b7bbae2cedd8" (UID: "09d50452-2c35-4ae2-931d-b7bbae2cedd8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:45:35.165352 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.165323 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d50452-2c35-4ae2-931d-b7bbae2cedd8-kube-api-access-mvmq9" (OuterVolumeSpecName: "kube-api-access-mvmq9") pod "09d50452-2c35-4ae2-931d-b7bbae2cedd8" (UID: "09d50452-2c35-4ae2-931d-b7bbae2cedd8"). InnerVolumeSpecName "kube-api-access-mvmq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:45:35.169033 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.168996 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-util" (OuterVolumeSpecName: "util") pod "09d50452-2c35-4ae2-931d-b7bbae2cedd8" (UID: "09d50452-2c35-4ae2-931d-b7bbae2cedd8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:45:35.263498 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.263468 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-util\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:45:35.263638 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.263502 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvmq9\" (UniqueName: \"kubernetes.io/projected/09d50452-2c35-4ae2-931d-b7bbae2cedd8-kube-api-access-mvmq9\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:45:35.263638 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.263517 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d50452-2c35-4ae2-931d-b7bbae2cedd8-bundle\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:45:35.890832 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.890790 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" event={"ID":"09d50452-2c35-4ae2-931d-b7bbae2cedd8","Type":"ContainerDied","Data":"606500274b3497f7632fc3085ee0b864c0050e0455f17f96933a679a6c096b14"} Apr 16 20:45:35.890832 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.890822 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f77c1a82ade775f7608969957d57ee0b8db93eeada9825bd6f7f7156c9n5khx" Apr 16 20:45:35.890832 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.890831 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="606500274b3497f7632fc3085ee0b864c0050e0455f17f96933a679a6c096b14" Apr 16 20:45:35.892272 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.892241 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" event={"ID":"edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc","Type":"ContainerStarted","Data":"7a832cf8497102447fd59210759ea5c8d56be9cee8ee78d419f1503fe3282103"} Apr 16 20:45:35.892408 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.892392 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:45:35.922336 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:35.922283 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" podStartSLOduration=1.4204763009999999 podStartE2EDuration="3.922266176s" podCreationTimestamp="2026-04-16 20:45:32 +0000 UTC" firstStartedPulling="2026-04-16 20:45:33.221782641 +0000 UTC m=+463.397146885" lastFinishedPulling="2026-04-16 20:45:35.723572513 +0000 UTC m=+465.898936760" observedRunningTime="2026-04-16 20:45:35.922093335 +0000 UTC m=+466.097457600" watchObservedRunningTime="2026-04-16 20:45:35.922266176 +0000 UTC m=+466.097630444" Apr 16 20:45:39.865213 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:39.865185 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5494fc4578-7c4xp" Apr 16 20:45:46.897445 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:45:46.897418 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7cd8df7dd5-t8gln" Apr 16 20:46:21.262870 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.262829 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d"] Apr 16 20:46:21.263371 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.263108 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d50452-2c35-4ae2-931d-b7bbae2cedd8" containerName="util" Apr 16 20:46:21.263371 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.263119 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d50452-2c35-4ae2-931d-b7bbae2cedd8" containerName="util" Apr 16 20:46:21.263371 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.263128 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d50452-2c35-4ae2-931d-b7bbae2cedd8" containerName="pull" Apr 16 20:46:21.263371 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.263135 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d50452-2c35-4ae2-931d-b7bbae2cedd8" containerName="pull" Apr 16 20:46:21.263371 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.263158 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d50452-2c35-4ae2-931d-b7bbae2cedd8" containerName="extract" Apr 16 20:46:21.263371 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.263164 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d50452-2c35-4ae2-931d-b7bbae2cedd8" containerName="extract" Apr 16 20:46:21.263371 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.263208 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="09d50452-2c35-4ae2-931d-b7bbae2cedd8" containerName="extract" Apr 16 20:46:21.265387 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.265366 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.268040 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.268018 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"istio-ca-root-cert\"" Apr 16 20:46:21.269013 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.268994 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-v476v\"" Apr 16 20:46:21.284281 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.284258 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d"] Apr 16 20:46:21.375500 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.375467 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.375654 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.375509 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.375654 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.375530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c514a92-cf85-4f01-866d-3d4033318cd4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.375654 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.375598 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.375654 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.375653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js95r\" (UniqueName: \"kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-kube-api-access-js95r\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.375870 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.375679 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.375870 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.375695 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.375870 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.375715 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.375870 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.375731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.476662 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.476631 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.476662 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.476664 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.476805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.476684 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.476805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.476700 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.476805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.476743 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.476805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.476768 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.476805 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.476793 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c514a92-cf85-4f01-866d-3d4033318cd4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.477036 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.476831 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.477036 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.476857 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-js95r\" (UniqueName: \"kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-kube-api-access-js95r\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.477150 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.477072 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.477207 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.477185 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.477263 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.477226 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.477338 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.477320 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.477578 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.477556 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c514a92-cf85-4f01-866d-3d4033318cd4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.478987 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.478961 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.479175 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.479156 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.488456 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.488431 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59"] Apr 16 20:46:21.490596 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.490579 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.498551 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.498524 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.503719 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.503702 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js95r\" (UniqueName: \"kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-kube-api-access-js95r\") pod \"data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.512783 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.512762 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59"] Apr 16 20:46:21.575722 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.575701 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:21.577396 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.577376 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.577457 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.577406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.577457 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.577426 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.577457 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.577443 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.577570 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.577499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.577570 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.577544 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.577570 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.577562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059cd12f-f5b5-444d-9497-f0e292f1ed45-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.577693 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.577623 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.577693 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.577676 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r449c\" (UniqueName: \"kubernetes.io/projected/059cd12f-f5b5-444d-9497-f0e292f1ed45-kube-api-access-r449c\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.679214 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679179 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.679340 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679222 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059cd12f-f5b5-444d-9497-f0e292f1ed45-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.679340 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679289 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.679340 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679323 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r449c\" (UniqueName: \"kubernetes.io/projected/059cd12f-f5b5-444d-9497-f0e292f1ed45-kube-api-access-r449c\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.679490 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679367 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.679490 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679396 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.679490 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679429 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.679490 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679458 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.679699 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679508 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.679864 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679826 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.680014 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.679995 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.680237 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.680167 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.680426 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.680398 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059cd12f-f5b5-444d-9497-f0e292f1ed45-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.680576 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.680551 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.681742 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.681722 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.682064 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.681938 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.703280 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.703246 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059cd12f-f5b5-444d-9497-f0e292f1ed45-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.704178 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.704156 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d"] Apr 16 20:46:21.704260 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.704188 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r449c\" (UniqueName: \"kubernetes.io/projected/059cd12f-f5b5-444d-9497-f0e292f1ed45-kube-api-access-r449c\") pod \"data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59\" (UID: \"059cd12f-f5b5-444d-9497-f0e292f1ed45\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.707591 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:46:21.707568 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c514a92_cf85_4f01_866d_3d4033318cd4.slice/crio-49f06fe6560784914b75115aa4f84488cbaf53fc743e92c1b0fb496bff546db7 WatchSource:0}: Error finding container 49f06fe6560784914b75115aa4f84488cbaf53fc743e92c1b0fb496bff546db7: Status 404 returned error can't find the container with id 49f06fe6560784914b75115aa4f84488cbaf53fc743e92c1b0fb496bff546db7 Apr 16 20:46:21.800496 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.800471 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:21.946005 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:21.945979 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59"] Apr 16 20:46:21.947969 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:46:21.947944 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059cd12f_f5b5_444d_9497_f0e292f1ed45.slice/crio-6e0944dbceab2feff81d308d7bdb38e7608726ef055899437e48491bd1a276ad WatchSource:0}: Error finding container 6e0944dbceab2feff81d308d7bdb38e7608726ef055899437e48491bd1a276ad: Status 404 returned error can't find the container with id 6e0944dbceab2feff81d308d7bdb38e7608726ef055899437e48491bd1a276ad Apr 16 20:46:22.035765 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:22.035731 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" event={"ID":"059cd12f-f5b5-444d-9497-f0e292f1ed45","Type":"ContainerStarted","Data":"6e0944dbceab2feff81d308d7bdb38e7608726ef055899437e48491bd1a276ad"} Apr 16 20:46:22.036753 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:22.036728 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" event={"ID":"9c514a92-cf85-4f01-866d-3d4033318cd4","Type":"ContainerStarted","Data":"49f06fe6560784914b75115aa4f84488cbaf53fc743e92c1b0fb496bff546db7"} Apr 16 20:46:24.576534 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:24.576486 2562 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Apr 16 20:46:24.576846 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:24.576567 2562 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Apr 16 20:46:24.576846 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:24.576597 2562 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Apr 16 20:46:24.583157 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:24.583129 2562 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Apr 16 20:46:24.583244 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:24.583186 2562 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Apr 16 20:46:24.583244 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:24.583216 2562 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Apr 16 20:46:25.051993 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:25.051959 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" event={"ID":"9c514a92-cf85-4f01-866d-3d4033318cd4","Type":"ContainerStarted","Data":"49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16"} Apr 16 20:46:25.053324 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:25.053302 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" event={"ID":"059cd12f-f5b5-444d-9497-f0e292f1ed45","Type":"ContainerStarted","Data":"4719a712107589e4c84ce480eb341e5077fee337180f26d04e7271a47b99b9ba"} Apr 16 20:46:25.071128 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:25.071087 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" podStartSLOduration=1.197467421 podStartE2EDuration="4.071075487s" podCreationTimestamp="2026-04-16 20:46:21 +0000 UTC" firstStartedPulling="2026-04-16 20:46:21.70933512 +0000 UTC m=+511.884699363" lastFinishedPulling="2026-04-16 20:46:24.582943186 +0000 UTC m=+514.758307429" observedRunningTime="2026-04-16 20:46:25.069264558 +0000 UTC m=+515.244628824" watchObservedRunningTime="2026-04-16 20:46:25.071075487 +0000 UTC m=+515.246439753" Apr 16 20:46:25.087991 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:25.087951 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" podStartSLOduration=1.461439208 podStartE2EDuration="4.08793947s" podCreationTimestamp="2026-04-16 20:46:21 +0000 UTC" firstStartedPulling="2026-04-16 20:46:21.949780391 +0000 UTC m=+512.125144635" lastFinishedPulling="2026-04-16 20:46:24.576280654 +0000 UTC m=+514.751644897" observedRunningTime="2026-04-16 20:46:25.086944709 +0000 UTC m=+515.262308976" watchObservedRunningTime="2026-04-16 20:46:25.08793947 +0000 UTC m=+515.263303736" Apr 16 20:46:25.576158 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:25.576135 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:25.577536 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:25.577514 2562 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Apr 16 20:46:25.577918 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:25.577578 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" podUID="9c514a92-cf85-4f01-866d-3d4033318cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Apr 16 20:46:25.800817 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:25.800793 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:25.805072 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:25.805046 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:26.063334 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:26.063305 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:26.064153 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:26.064132 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59" Apr 16 20:46:26.110509 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:26.110482 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d"] Apr 16 20:46:26.576753 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:26.576727 2562 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Apr 16 20:46:26.576955 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:26.576783 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" podUID="9c514a92-cf85-4f01-866d-3d4033318cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Apr 16 20:46:27.576854 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:27.576817 2562 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Apr 16 20:46:27.577299 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:27.576882 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" podUID="9c514a92-cf85-4f01-866d-3d4033318cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Apr 16 20:46:28.069504 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:28.069472 2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" podUID="9c514a92-cf85-4f01-866d-3d4033318cd4" containerName="istio-proxy" containerID="cri-o://49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16" gracePeriod=30 Apr 16 20:46:33.301325 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.301305 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:33.466258 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466192 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-envoy\") pod \"9c514a92-cf85-4f01-866d-3d4033318cd4\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " Apr 16 20:46:33.466258 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466221 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-data\") pod \"9c514a92-cf85-4f01-866d-3d4033318cd4\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " Apr 16 20:46:33.466258 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466246 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js95r\" (UniqueName: \"kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-kube-api-access-js95r\") pod \"9c514a92-cf85-4f01-866d-3d4033318cd4\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " Apr 16 20:46:33.466500 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466271 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-token\") pod \"9c514a92-cf85-4f01-866d-3d4033318cd4\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " Apr 16 20:46:33.466500 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466286 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-socket\") pod \"9c514a92-cf85-4f01-866d-3d4033318cd4\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " Apr 16 20:46:33.466500 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466308 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-podinfo\") pod \"9c514a92-cf85-4f01-866d-3d4033318cd4\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " Apr 16 20:46:33.466500 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466335 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c514a92-cf85-4f01-866d-3d4033318cd4-istiod-ca-cert\") pod \"9c514a92-cf85-4f01-866d-3d4033318cd4\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " Apr 16 20:46:33.466500 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466359 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-credential-socket\") pod \"9c514a92-cf85-4f01-866d-3d4033318cd4\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " Apr 16 20:46:33.466500 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466435 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-certs\") pod \"9c514a92-cf85-4f01-866d-3d4033318cd4\" (UID: \"9c514a92-cf85-4f01-866d-3d4033318cd4\") " Apr 16 20:46:33.466841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466538 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c514a92-cf85-4f01-866d-3d4033318cd4" (UID: "9c514a92-cf85-4f01-866d-3d4033318cd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:46:33.466841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466596 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c514a92-cf85-4f01-866d-3d4033318cd4" (UID: "9c514a92-cf85-4f01-866d-3d4033318cd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:46:33.466841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466749 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-data\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:46:33.466841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466764 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c514a92-cf85-4f01-866d-3d4033318cd4" (UID: "9c514a92-cf85-4f01-866d-3d4033318cd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:46:33.466841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466769 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-socket\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:46:33.466841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466822 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c514a92-cf85-4f01-866d-3d4033318cd4" (UID: "9c514a92-cf85-4f01-866d-3d4033318cd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:46:33.466841 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.466830 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c514a92-cf85-4f01-866d-3d4033318cd4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c514a92-cf85-4f01-866d-3d4033318cd4" (UID: "9c514a92-cf85-4f01-866d-3d4033318cd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:46:33.468677 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.468587 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c514a92-cf85-4f01-866d-3d4033318cd4" (UID: "9c514a92-cf85-4f01-866d-3d4033318cd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:46:33.468677 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.468626 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c514a92-cf85-4f01-866d-3d4033318cd4" (UID: "9c514a92-cf85-4f01-866d-3d4033318cd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Apr 16 20:46:33.468819 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.468723 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c514a92-cf85-4f01-866d-3d4033318cd4" (UID: "9c514a92-cf85-4f01-866d-3d4033318cd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:46:33.468819 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.468737 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-kube-api-access-js95r" (OuterVolumeSpecName: "kube-api-access-js95r") pod "9c514a92-cf85-4f01-866d-3d4033318cd4" (UID: "9c514a92-cf85-4f01-866d-3d4033318cd4"). InnerVolumeSpecName "kube-api-access-js95r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:46:33.567788 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.567763 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-envoy\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:46:33.567788 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.567788 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js95r\" (UniqueName: \"kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-kube-api-access-js95r\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:46:33.567920 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.567802 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-token\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:46:33.567920 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.567817 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c514a92-cf85-4f01-866d-3d4033318cd4-istio-podinfo\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:46:33.567920 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.567829 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c514a92-cf85-4f01-866d-3d4033318cd4-istiod-ca-cert\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:46:33.567920 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.567842 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-credential-socket\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:46:33.567920 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:33.567857 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c514a92-cf85-4f01-866d-3d4033318cd4-workload-certs\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:46:34.088465 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:34.088433 2562 generic.go:358] "Generic (PLEG): container finished" podID="9c514a92-cf85-4f01-866d-3d4033318cd4" containerID="49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16" exitCode=0 Apr 16 20:46:34.088661 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:34.088503 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" Apr 16 20:46:34.088661 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:34.088505 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" event={"ID":"9c514a92-cf85-4f01-866d-3d4033318cd4","Type":"ContainerDied","Data":"49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16"} Apr 16 20:46:34.088661 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:34.088626 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d" event={"ID":"9c514a92-cf85-4f01-866d-3d4033318cd4","Type":"ContainerDied","Data":"49f06fe6560784914b75115aa4f84488cbaf53fc743e92c1b0fb496bff546db7"} Apr 16 20:46:34.088661 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:34.088641 2562 scope.go:117] "RemoveContainer" containerID="49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16" Apr 16 20:46:34.097013 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:34.096986 2562 scope.go:117] "RemoveContainer" containerID="49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16" Apr 16 20:46:34.097270 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:46:34.097251 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16\": container with ID starting with 49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16 not found: ID does not exist" containerID="49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16" Apr 16 20:46:34.097323 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:34.097278 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16"} err="failed to get container status \"49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16\": rpc error: code = NotFound desc = could not find container \"49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16\": container with ID starting with 49f1cb28f3db259e15e71ce76b742400a1841cabf30122830631b7e780b1da16 not found: ID does not exist" Apr 16 20:46:34.109523 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:34.109497 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d"] Apr 16 20:46:34.112768 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:34.112741 2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d9c6bcd5vdb2d"] Apr 16 20:46:34.502389 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:34.502312 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c514a92-cf85-4f01-866d-3d4033318cd4" path="/var/lib/kubelet/pods/9c514a92-cf85-4f01-866d-3d4033318cd4/volumes" Apr 16 20:46:50.840738 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:50.840708 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c956p"] Apr 16 20:46:50.841086 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:50.841002 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c514a92-cf85-4f01-866d-3d4033318cd4" containerName="istio-proxy" Apr 16 20:46:50.841086 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:50.841014 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c514a92-cf85-4f01-866d-3d4033318cd4" containerName="istio-proxy" Apr 16 20:46:50.841086 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:50.841071 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="9c514a92-cf85-4f01-866d-3d4033318cd4" containerName="istio-proxy" Apr 16 20:46:50.846022 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:50.845999 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c956p" Apr 16 20:46:50.848773 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:50.848751 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Apr 16 20:46:50.850008 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:50.849992 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-xj86m\"" Apr 16 20:46:50.850008 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:50.850005 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Apr 16 20:46:50.854564 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:50.854538 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c956p"] Apr 16 20:46:50.992156 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:50.992116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prsfq\" (UniqueName: \"kubernetes.io/projected/d25dab51-5ee8-4367-98fc-4c911e21c229-kube-api-access-prsfq\") pod \"kuadrant-operator-catalog-c956p\" (UID: \"d25dab51-5ee8-4367-98fc-4c911e21c229\") " pod="kuadrant-system/kuadrant-operator-catalog-c956p" Apr 16 20:46:51.093304 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.093237 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-prsfq\" (UniqueName: \"kubernetes.io/projected/d25dab51-5ee8-4367-98fc-4c911e21c229-kube-api-access-prsfq\") pod \"kuadrant-operator-catalog-c956p\" (UID: \"d25dab51-5ee8-4367-98fc-4c911e21c229\") " pod="kuadrant-system/kuadrant-operator-catalog-c956p" Apr 16 20:46:51.101410 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.101379 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prsfq\" (UniqueName: \"kubernetes.io/projected/d25dab51-5ee8-4367-98fc-4c911e21c229-kube-api-access-prsfq\") pod \"kuadrant-operator-catalog-c956p\" (UID: \"d25dab51-5ee8-4367-98fc-4c911e21c229\") " pod="kuadrant-system/kuadrant-operator-catalog-c956p" Apr 16 20:46:51.156470 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.156447 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c956p" Apr 16 20:46:51.202431 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.202384 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c956p"] Apr 16 20:46:51.273783 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.273755 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c956p"] Apr 16 20:46:51.276295 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:46:51.276268 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25dab51_5ee8_4367_98fc_4c911e21c229.slice/crio-3cc107e5b4479d2dd9503c1ad05cf2acd40c09d2b72dfa6422d1478d2c02fc2f WatchSource:0}: Error finding container 3cc107e5b4479d2dd9503c1ad05cf2acd40c09d2b72dfa6422d1478d2c02fc2f: Status 404 returned error can't find the container with id 3cc107e5b4479d2dd9503c1ad05cf2acd40c09d2b72dfa6422d1478d2c02fc2f Apr 16 20:46:51.411184 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.411128 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d6c2b"] Apr 16 20:46:51.413836 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.413821 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" Apr 16 20:46:51.421177 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.421157 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d6c2b"] Apr 16 20:46:51.597490 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.597465 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rb6b\" (UniqueName: \"kubernetes.io/projected/3e772558-6975-4286-a0a0-4a7c45ecc923-kube-api-access-9rb6b\") pod \"kuadrant-operator-catalog-d6c2b\" (UID: \"3e772558-6975-4286-a0a0-4a7c45ecc923\") " pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" Apr 16 20:46:51.698043 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.697979 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9rb6b\" (UniqueName: \"kubernetes.io/projected/3e772558-6975-4286-a0a0-4a7c45ecc923-kube-api-access-9rb6b\") pod \"kuadrant-operator-catalog-d6c2b\" (UID: \"3e772558-6975-4286-a0a0-4a7c45ecc923\") " pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" Apr 16 20:46:51.706003 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.705980 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rb6b\" (UniqueName: \"kubernetes.io/projected/3e772558-6975-4286-a0a0-4a7c45ecc923-kube-api-access-9rb6b\") pod \"kuadrant-operator-catalog-d6c2b\" (UID: \"3e772558-6975-4286-a0a0-4a7c45ecc923\") " pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" Apr 16 20:46:51.723910 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.723881 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" Apr 16 20:46:51.850681 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:51.850646 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d6c2b"] Apr 16 20:46:51.868549 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:46:51.868520 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e772558_6975_4286_a0a0_4a7c45ecc923.slice/crio-ecaab30bfb39b886475d6c68a6f4f63bd4490e7bbbde0effa794c1f7cdf8c583 WatchSource:0}: Error finding container ecaab30bfb39b886475d6c68a6f4f63bd4490e7bbbde0effa794c1f7cdf8c583: Status 404 returned error can't find the container with id ecaab30bfb39b886475d6c68a6f4f63bd4490e7bbbde0effa794c1f7cdf8c583 Apr 16 20:46:52.146588 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:52.146554 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" event={"ID":"3e772558-6975-4286-a0a0-4a7c45ecc923","Type":"ContainerStarted","Data":"ecaab30bfb39b886475d6c68a6f4f63bd4490e7bbbde0effa794c1f7cdf8c583"} Apr 16 20:46:52.147771 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:52.147744 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c956p" event={"ID":"d25dab51-5ee8-4367-98fc-4c911e21c229","Type":"ContainerStarted","Data":"3cc107e5b4479d2dd9503c1ad05cf2acd40c09d2b72dfa6422d1478d2c02fc2f"} Apr 16 20:46:54.155505 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:54.155471 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c956p" event={"ID":"d25dab51-5ee8-4367-98fc-4c911e21c229","Type":"ContainerStarted","Data":"85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f"} Apr 16 20:46:54.155950 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:54.155545 2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-c956p" podUID="d25dab51-5ee8-4367-98fc-4c911e21c229" containerName="registry-server" containerID="cri-o://85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f" gracePeriod=2 Apr 16 20:46:54.156845 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:54.156816 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" event={"ID":"3e772558-6975-4286-a0a0-4a7c45ecc923","Type":"ContainerStarted","Data":"f03cc3c88db6b904d0ea9b66bacfcf561146e0056f0a5e403b2904213c54c856"} Apr 16 20:46:54.171860 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:54.171821 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-c956p" podStartSLOduration=1.904542603 podStartE2EDuration="4.171807165s" podCreationTimestamp="2026-04-16 20:46:50 +0000 UTC" firstStartedPulling="2026-04-16 20:46:51.277709005 +0000 UTC m=+541.453073266" lastFinishedPulling="2026-04-16 20:46:53.544973581 +0000 UTC m=+543.720337828" observedRunningTime="2026-04-16 20:46:54.170265055 +0000 UTC m=+544.345629322" watchObservedRunningTime="2026-04-16 20:46:54.171807165 +0000 UTC m=+544.347171432" Apr 16 20:46:54.187477 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:54.187440 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" podStartSLOduration=1.517937098 podStartE2EDuration="3.18742967s" podCreationTimestamp="2026-04-16 20:46:51 +0000 UTC" firstStartedPulling="2026-04-16 20:46:51.870009278 +0000 UTC m=+542.045373522" lastFinishedPulling="2026-04-16 20:46:53.539501843 +0000 UTC m=+543.714866094" observedRunningTime="2026-04-16 20:46:54.186190657 +0000 UTC m=+544.361554922" watchObservedRunningTime="2026-04-16 20:46:54.18742967 +0000 UTC m=+544.362793936" Apr 16 20:46:54.387582 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:54.387560 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c956p" Apr 16 20:46:54.522807 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:54.522730 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prsfq\" (UniqueName: \"kubernetes.io/projected/d25dab51-5ee8-4367-98fc-4c911e21c229-kube-api-access-prsfq\") pod \"d25dab51-5ee8-4367-98fc-4c911e21c229\" (UID: \"d25dab51-5ee8-4367-98fc-4c911e21c229\") " Apr 16 20:46:54.525006 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:54.524972 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25dab51-5ee8-4367-98fc-4c911e21c229-kube-api-access-prsfq" (OuterVolumeSpecName: "kube-api-access-prsfq") pod "d25dab51-5ee8-4367-98fc-4c911e21c229" (UID: "d25dab51-5ee8-4367-98fc-4c911e21c229"). InnerVolumeSpecName "kube-api-access-prsfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:46:54.623515 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:54.623491 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prsfq\" (UniqueName: \"kubernetes.io/projected/d25dab51-5ee8-4367-98fc-4c911e21c229-kube-api-access-prsfq\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:46:55.161121 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:55.161086 2562 generic.go:358] "Generic (PLEG): container finished" podID="d25dab51-5ee8-4367-98fc-4c911e21c229" containerID="85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f" exitCode=0 Apr 16 20:46:55.161516 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:55.161182 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c956p" Apr 16 20:46:55.161516 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:55.161177 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c956p" event={"ID":"d25dab51-5ee8-4367-98fc-4c911e21c229","Type":"ContainerDied","Data":"85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f"} Apr 16 20:46:55.161516 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:55.161291 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c956p" event={"ID":"d25dab51-5ee8-4367-98fc-4c911e21c229","Type":"ContainerDied","Data":"3cc107e5b4479d2dd9503c1ad05cf2acd40c09d2b72dfa6422d1478d2c02fc2f"} Apr 16 20:46:55.161516 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:55.161310 2562 scope.go:117] "RemoveContainer" containerID="85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f" Apr 16 20:46:55.170683 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:55.170666 2562 scope.go:117] "RemoveContainer" containerID="85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f" Apr 16 20:46:55.170928 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:46:55.170910 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f\": container with ID starting with 85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f not found: ID does not exist" containerID="85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f" Apr 16 20:46:55.171002 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:55.170936 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f"} err="failed to get container status \"85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f\": rpc error: code = NotFound desc = could not find container \"85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f\": container with ID starting with 85dbf716807cb3e247b7309c640c3a73baa56adad940d4f720071f71142b7e9f not found: ID does not exist" Apr 16 20:46:55.181767 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:55.181747 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c956p"] Apr 16 20:46:55.187575 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:55.187556 2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c956p"] Apr 16 20:46:56.501531 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:46:56.501497 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25dab51-5ee8-4367-98fc-4c911e21c229" path="/var/lib/kubelet/pods/d25dab51-5ee8-4367-98fc-4c911e21c229/volumes" Apr 16 20:47:01.724784 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:01.724743 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" Apr 16 20:47:01.725276 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:01.724811 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" Apr 16 20:47:01.745712 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:01.745688 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" Apr 16 20:47:02.202521 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:02.202490 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-d6c2b" Apr 16 20:47:06.446141 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.446105 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq"] Apr 16 20:47:06.446502 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.446392 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25dab51-5ee8-4367-98fc-4c911e21c229" containerName="registry-server" Apr 16 20:47:06.446502 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.446403 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25dab51-5ee8-4367-98fc-4c911e21c229" containerName="registry-server" Apr 16 20:47:06.446502 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.446451 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="d25dab51-5ee8-4367-98fc-4c911e21c229" containerName="registry-server" Apr 16 20:47:06.451068 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.451050 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.453639 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.453619 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c6jk2\"" Apr 16 20:47:06.460478 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.460456 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq"] Apr 16 20:47:06.503815 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.503795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.503934 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.503916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.503980 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.503950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29vx\" (UniqueName: \"kubernetes.io/projected/dace4755-9fcf-42d7-9ab4-e3a9d371f004-kube-api-access-d29vx\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.604248 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.604228 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.604355 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.604255 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d29vx\" (UniqueName: \"kubernetes.io/projected/dace4755-9fcf-42d7-9ab4-e3a9d371f004-kube-api-access-d29vx\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.604355 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.604278 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.604555 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.604538 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.604704 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.604685 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.612848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.612825 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29vx\" (UniqueName: \"kubernetes.io/projected/dace4755-9fcf-42d7-9ab4-e3a9d371f004-kube-api-access-d29vx\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.760091 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.760026 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:06.843796 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.843720 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm"] Apr 16 20:47:06.848627 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.848590 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:06.854941 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.854907 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm"] Apr 16 20:47:06.880332 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.880313 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq"] Apr 16 20:47:06.881964 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:47:06.881939 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddace4755_9fcf_42d7_9ab4_e3a9d371f004.slice/crio-c62c989cc36b22b998740aa31fb3506f2826897259592f966cc45801dec4ea4b WatchSource:0}: Error finding container c62c989cc36b22b998740aa31fb3506f2826897259592f966cc45801dec4ea4b: Status 404 returned error can't find the container with id c62c989cc36b22b998740aa31fb3506f2826897259592f966cc45801dec4ea4b Apr 16 20:47:06.905877 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.905854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:06.905978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.905908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:06.905978 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:06.905952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmtk\" (UniqueName: \"kubernetes.io/projected/b5808b89-67e6-4adf-886b-f3ff4499a1ff-kube-api-access-fgmtk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:07.007073 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.007042 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:07.007165 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.007089 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:07.007212 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.007192 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmtk\" (UniqueName: \"kubernetes.io/projected/b5808b89-67e6-4adf-886b-f3ff4499a1ff-kube-api-access-fgmtk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:07.007357 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.007342 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:07.007418 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.007398 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:07.015087 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.015045 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmtk\" (UniqueName: \"kubernetes.io/projected/b5808b89-67e6-4adf-886b-f3ff4499a1ff-kube-api-access-fgmtk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:07.160614 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.160579 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:07.210397 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.210371 2562 generic.go:358] "Generic (PLEG): container finished" podID="dace4755-9fcf-42d7-9ab4-e3a9d371f004" containerID="5ebbf8e5affb3eba9791fa2903e5b29ef57eaf5dbc990356f5f6a9b3197eafc0" exitCode=0 Apr 16 20:47:07.210513 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.210475 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" event={"ID":"dace4755-9fcf-42d7-9ab4-e3a9d371f004","Type":"ContainerDied","Data":"5ebbf8e5affb3eba9791fa2903e5b29ef57eaf5dbc990356f5f6a9b3197eafc0"} Apr 16 20:47:07.210513 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.210506 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" event={"ID":"dace4755-9fcf-42d7-9ab4-e3a9d371f004","Type":"ContainerStarted","Data":"c62c989cc36b22b998740aa31fb3506f2826897259592f966cc45801dec4ea4b"} Apr 16 20:47:07.246322 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.246283 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk"] Apr 16 20:47:07.251409 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.251389 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.257103 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.257075 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk"] Apr 16 20:47:07.288909 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.288886 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm"] Apr 16 20:47:07.291112 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:47:07.291086 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5808b89_67e6_4adf_886b_f3ff4499a1ff.slice/crio-57875dd8b98d91af00f5c7946deca805c2eacbc848d28739e2783a1cb4b1c88e WatchSource:0}: Error finding container 57875dd8b98d91af00f5c7946deca805c2eacbc848d28739e2783a1cb4b1c88e: Status 404 returned error can't find the container with id 57875dd8b98d91af00f5c7946deca805c2eacbc848d28739e2783a1cb4b1c88e Apr 16 20:47:07.309081 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.309054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl52p\" (UniqueName: \"kubernetes.io/projected/53bbc7b6-a3e5-410e-9106-08828d972dee-kube-api-access-hl52p\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.309194 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.309091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.309263 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.309246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.410795 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.410765 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.410929 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.410826 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hl52p\" (UniqueName: \"kubernetes.io/projected/53bbc7b6-a3e5-410e-9106-08828d972dee-kube-api-access-hl52p\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.410929 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.410873 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.411193 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.411168 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.411193 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.411187 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.419373 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.419351 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl52p\" (UniqueName: \"kubernetes.io/projected/53bbc7b6-a3e5-410e-9106-08828d972dee-kube-api-access-hl52p\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.563381 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.563363 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:07.648548 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.648522 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm"] Apr 16 20:47:07.653273 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.653255 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:07.659145 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.659105 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm"] Apr 16 20:47:07.683111 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.683058 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk"] Apr 16 20:47:07.685643 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:47:07.685618 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bbc7b6_a3e5_410e_9106_08828d972dee.slice/crio-111479edb3426ff10b83c08690ec3bd49d43a6d871f1c60ead7d3f35b5d566f8 WatchSource:0}: Error finding container 111479edb3426ff10b83c08690ec3bd49d43a6d871f1c60ead7d3f35b5d566f8: Status 404 returned error can't find the container with id 111479edb3426ff10b83c08690ec3bd49d43a6d871f1c60ead7d3f35b5d566f8 Apr 16 20:47:07.714626 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.714586 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:07.714711 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.714643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9dg\" (UniqueName: \"kubernetes.io/projected/2dc44f77-16a9-4713-b59d-c07093e1b1f0-kube-api-access-2c9dg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:07.714751 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.714713 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:07.815417 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.815363 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:07.815417 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.815410 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9dg\" (UniqueName: \"kubernetes.io/projected/2dc44f77-16a9-4713-b59d-c07093e1b1f0-kube-api-access-2c9dg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:07.815548 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.815456 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:07.815720 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.815703 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:07.815811 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.815794 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:07.823325 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.823302 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9dg\" (UniqueName: \"kubernetes.io/projected/2dc44f77-16a9-4713-b59d-c07093e1b1f0-kube-api-access-2c9dg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:07.964533 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:07.964499 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:08.135045 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:08.135017 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm"] Apr 16 20:47:08.136862 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:47:08.136833 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc44f77_16a9_4713_b59d_c07093e1b1f0.slice/crio-537ac8cf09ccf254fdff50025dad948c1d426a82e2d57a2504b57b4c028c5e58 WatchSource:0}: Error finding container 537ac8cf09ccf254fdff50025dad948c1d426a82e2d57a2504b57b4c028c5e58: Status 404 returned error can't find the container with id 537ac8cf09ccf254fdff50025dad948c1d426a82e2d57a2504b57b4c028c5e58 Apr 16 20:47:08.214295 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:08.214265 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" event={"ID":"2dc44f77-16a9-4713-b59d-c07093e1b1f0","Type":"ContainerStarted","Data":"537ac8cf09ccf254fdff50025dad948c1d426a82e2d57a2504b57b4c028c5e58"} Apr 16 20:47:08.215426 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:08.215404 2562 generic.go:358] "Generic (PLEG): container finished" podID="b5808b89-67e6-4adf-886b-f3ff4499a1ff" containerID="a6fe2aa429fc13a7fcde51de34fc011a2eb4bb955ab39bfc8dcdfb2e021ce765" exitCode=0 Apr 16 20:47:08.215535 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:08.215483 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" event={"ID":"b5808b89-67e6-4adf-886b-f3ff4499a1ff","Type":"ContainerDied","Data":"a6fe2aa429fc13a7fcde51de34fc011a2eb4bb955ab39bfc8dcdfb2e021ce765"} Apr 16 20:47:08.215535 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:08.215518 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" event={"ID":"b5808b89-67e6-4adf-886b-f3ff4499a1ff","Type":"ContainerStarted","Data":"57875dd8b98d91af00f5c7946deca805c2eacbc848d28739e2783a1cb4b1c88e"} Apr 16 20:47:08.216894 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:08.216873 2562 generic.go:358] "Generic (PLEG): container finished" podID="53bbc7b6-a3e5-410e-9106-08828d972dee" containerID="7efa3816e5edc43dbade42eda4bd011c15d904b0a2cc19282a166087d5fccd18" exitCode=0 Apr 16 20:47:08.217002 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:08.216942 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" event={"ID":"53bbc7b6-a3e5-410e-9106-08828d972dee","Type":"ContainerDied","Data":"7efa3816e5edc43dbade42eda4bd011c15d904b0a2cc19282a166087d5fccd18"} Apr 16 20:47:08.217002 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:08.216968 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" event={"ID":"53bbc7b6-a3e5-410e-9106-08828d972dee","Type":"ContainerStarted","Data":"111479edb3426ff10b83c08690ec3bd49d43a6d871f1c60ead7d3f35b5d566f8"} Apr 16 20:47:08.218647 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:08.218623 2562 generic.go:358] "Generic (PLEG): container finished" podID="dace4755-9fcf-42d7-9ab4-e3a9d371f004" containerID="6687acd24e1845fc482af21c03a8de117a60960cf48065d04d69abc4123fbcd1" exitCode=0 Apr 16 20:47:08.218738 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:08.218669 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" event={"ID":"dace4755-9fcf-42d7-9ab4-e3a9d371f004","Type":"ContainerDied","Data":"6687acd24e1845fc482af21c03a8de117a60960cf48065d04d69abc4123fbcd1"} Apr 16 20:47:09.222887 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:09.222857 2562 generic.go:358] "Generic (PLEG): container finished" podID="2dc44f77-16a9-4713-b59d-c07093e1b1f0" containerID="02bbbd6cba78553dc9bc8720704ee4b7755e68ed2686a8f9044c982c7f5bf5ee" exitCode=0 Apr 16 20:47:09.223203 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:09.222933 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" event={"ID":"2dc44f77-16a9-4713-b59d-c07093e1b1f0","Type":"ContainerDied","Data":"02bbbd6cba78553dc9bc8720704ee4b7755e68ed2686a8f9044c982c7f5bf5ee"} Apr 16 20:47:09.224664 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:09.224641 2562 generic.go:358] "Generic (PLEG): container finished" podID="53bbc7b6-a3e5-410e-9106-08828d972dee" containerID="c47ca76165e8ec18acdbe086f2d71222528a92a1786b3cd6fd0089a4f7c9d800" exitCode=0 Apr 16 20:47:09.224752 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:09.224717 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" event={"ID":"53bbc7b6-a3e5-410e-9106-08828d972dee","Type":"ContainerDied","Data":"c47ca76165e8ec18acdbe086f2d71222528a92a1786b3cd6fd0089a4f7c9d800"} Apr 16 20:47:09.229771 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:09.229751 2562 generic.go:358] "Generic (PLEG): container finished" podID="dace4755-9fcf-42d7-9ab4-e3a9d371f004" containerID="42e2d514f1686aa90502a4676bd700227f0895ffb4718b326d5523701820d185" exitCode=0 Apr 16 20:47:09.229852 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:09.229791 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" event={"ID":"dace4755-9fcf-42d7-9ab4-e3a9d371f004","Type":"ContainerDied","Data":"42e2d514f1686aa90502a4676bd700227f0895ffb4718b326d5523701820d185"} Apr 16 20:47:10.241038 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.240960 2562 generic.go:358] "Generic (PLEG): container finished" podID="53bbc7b6-a3e5-410e-9106-08828d972dee" containerID="4ff706492c28c652456b7d844a91243bbeef588231f29ea771e22340a70cb143" exitCode=0 Apr 16 20:47:10.241038 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.241012 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" event={"ID":"53bbc7b6-a3e5-410e-9106-08828d972dee","Type":"ContainerDied","Data":"4ff706492c28c652456b7d844a91243bbeef588231f29ea771e22340a70cb143"} Apr 16 20:47:10.242685 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.242661 2562 generic.go:358] "Generic (PLEG): container finished" podID="2dc44f77-16a9-4713-b59d-c07093e1b1f0" containerID="7bcf2384796f106d0422878ddd3f2004ba58c169c5c07e10c89032623bdac4a8" exitCode=0 Apr 16 20:47:10.242804 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.242741 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" event={"ID":"2dc44f77-16a9-4713-b59d-c07093e1b1f0","Type":"ContainerDied","Data":"7bcf2384796f106d0422878ddd3f2004ba58c169c5c07e10c89032623bdac4a8"} Apr 16 20:47:10.244279 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.244257 2562 generic.go:358] "Generic (PLEG): container finished" podID="b5808b89-67e6-4adf-886b-f3ff4499a1ff" containerID="81cda81f79ca53c14a98af66932ef24372e4388aaa6f619fb663dc1e4806e30d" exitCode=0 Apr 16 20:47:10.244374 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.244285 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" event={"ID":"b5808b89-67e6-4adf-886b-f3ff4499a1ff","Type":"ContainerDied","Data":"81cda81f79ca53c14a98af66932ef24372e4388aaa6f619fb663dc1e4806e30d"} Apr 16 20:47:10.365851 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.365827 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:10.439850 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.439820 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-util\") pod \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " Apr 16 20:47:10.439967 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.439910 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-bundle\") pod \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " Apr 16 20:47:10.439967 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.439946 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d29vx\" (UniqueName: \"kubernetes.io/projected/dace4755-9fcf-42d7-9ab4-e3a9d371f004-kube-api-access-d29vx\") pod \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\" (UID: \"dace4755-9fcf-42d7-9ab4-e3a9d371f004\") " Apr 16 20:47:10.440339 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.440319 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-bundle" (OuterVolumeSpecName: "bundle") pod "dace4755-9fcf-42d7-9ab4-e3a9d371f004" (UID: "dace4755-9fcf-42d7-9ab4-e3a9d371f004"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:47:10.441910 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.441887 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dace4755-9fcf-42d7-9ab4-e3a9d371f004-kube-api-access-d29vx" (OuterVolumeSpecName: "kube-api-access-d29vx") pod "dace4755-9fcf-42d7-9ab4-e3a9d371f004" (UID: "dace4755-9fcf-42d7-9ab4-e3a9d371f004"). InnerVolumeSpecName "kube-api-access-d29vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:47:10.444812 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.444777 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-util" (OuterVolumeSpecName: "util") pod "dace4755-9fcf-42d7-9ab4-e3a9d371f004" (UID: "dace4755-9fcf-42d7-9ab4-e3a9d371f004"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:47:10.540668 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.540644 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-bundle\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:10.540668 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.540670 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d29vx\" (UniqueName: \"kubernetes.io/projected/dace4755-9fcf-42d7-9ab4-e3a9d371f004-kube-api-access-d29vx\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:10.540851 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:10.540685 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dace4755-9fcf-42d7-9ab4-e3a9d371f004-util\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:11.248984 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.248951 2562 generic.go:358] "Generic (PLEG): container finished" podID="2dc44f77-16a9-4713-b59d-c07093e1b1f0" containerID="3908f74f090b8289388058c052183fb4cb2ffa26bbc9ad8ddec925f1062d1f16" exitCode=0 Apr 16 20:47:11.249417 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.249035 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" event={"ID":"2dc44f77-16a9-4713-b59d-c07093e1b1f0","Type":"ContainerDied","Data":"3908f74f090b8289388058c052183fb4cb2ffa26bbc9ad8ddec925f1062d1f16"} Apr 16 20:47:11.250878 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.250852 2562 generic.go:358] "Generic (PLEG): container finished" podID="b5808b89-67e6-4adf-886b-f3ff4499a1ff" containerID="f70e1a12723142fd4d99dd57f8260d8efaebc45b85ea164f56ad686b2f62eee7" exitCode=0 Apr 16 20:47:11.250976 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.250916 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" event={"ID":"b5808b89-67e6-4adf-886b-f3ff4499a1ff","Type":"ContainerDied","Data":"f70e1a12723142fd4d99dd57f8260d8efaebc45b85ea164f56ad686b2f62eee7"} Apr 16 20:47:11.252486 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.252461 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" event={"ID":"dace4755-9fcf-42d7-9ab4-e3a9d371f004","Type":"ContainerDied","Data":"c62c989cc36b22b998740aa31fb3506f2826897259592f966cc45801dec4ea4b"} Apr 16 20:47:11.252577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.252491 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c62c989cc36b22b998740aa31fb3506f2826897259592f966cc45801dec4ea4b" Apr 16 20:47:11.252577 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.252500 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq" Apr 16 20:47:11.381179 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.381158 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:11.449261 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.449235 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl52p\" (UniqueName: \"kubernetes.io/projected/53bbc7b6-a3e5-410e-9106-08828d972dee-kube-api-access-hl52p\") pod \"53bbc7b6-a3e5-410e-9106-08828d972dee\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " Apr 16 20:47:11.449393 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.449301 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-util\") pod \"53bbc7b6-a3e5-410e-9106-08828d972dee\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " Apr 16 20:47:11.449393 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.449333 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-bundle\") pod \"53bbc7b6-a3e5-410e-9106-08828d972dee\" (UID: \"53bbc7b6-a3e5-410e-9106-08828d972dee\") " Apr 16 20:47:11.449924 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.449900 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-bundle" (OuterVolumeSpecName: "bundle") pod "53bbc7b6-a3e5-410e-9106-08828d972dee" (UID: "53bbc7b6-a3e5-410e-9106-08828d972dee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:47:11.451228 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.451206 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bbc7b6-a3e5-410e-9106-08828d972dee-kube-api-access-hl52p" (OuterVolumeSpecName: "kube-api-access-hl52p") pod "53bbc7b6-a3e5-410e-9106-08828d972dee" (UID: "53bbc7b6-a3e5-410e-9106-08828d972dee"). InnerVolumeSpecName "kube-api-access-hl52p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:47:11.457103 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.457080 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-util" (OuterVolumeSpecName: "util") pod "53bbc7b6-a3e5-410e-9106-08828d972dee" (UID: "53bbc7b6-a3e5-410e-9106-08828d972dee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:47:11.550437 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.550416 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl52p\" (UniqueName: \"kubernetes.io/projected/53bbc7b6-a3e5-410e-9106-08828d972dee-kube-api-access-hl52p\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:11.550507 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.550437 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-util\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:11.550507 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:11.550447 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53bbc7b6-a3e5-410e-9106-08828d972dee-bundle\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:12.256853 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.256826 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" Apr 16 20:47:12.256853 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.256835 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk" event={"ID":"53bbc7b6-a3e5-410e-9106-08828d972dee","Type":"ContainerDied","Data":"111479edb3426ff10b83c08690ec3bd49d43a6d871f1c60ead7d3f35b5d566f8"} Apr 16 20:47:12.257245 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.256865 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="111479edb3426ff10b83c08690ec3bd49d43a6d871f1c60ead7d3f35b5d566f8" Apr 16 20:47:12.383303 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.383280 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:12.412308 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.412287 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:12.456452 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.456427 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c9dg\" (UniqueName: \"kubernetes.io/projected/2dc44f77-16a9-4713-b59d-c07093e1b1f0-kube-api-access-2c9dg\") pod \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " Apr 16 20:47:12.456575 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.456470 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-util\") pod \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " Apr 16 20:47:12.456575 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.456490 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-util\") pod \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " Apr 16 20:47:12.456575 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.456515 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgmtk\" (UniqueName: \"kubernetes.io/projected/b5808b89-67e6-4adf-886b-f3ff4499a1ff-kube-api-access-fgmtk\") pod \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " Apr 16 20:47:12.456575 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.456557 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-bundle\") pod \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\" (UID: \"b5808b89-67e6-4adf-886b-f3ff4499a1ff\") " Apr 16 20:47:12.456808 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.456587 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-bundle\") pod \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\" (UID: \"2dc44f77-16a9-4713-b59d-c07093e1b1f0\") " Apr 16 20:47:12.457247 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.457218 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-bundle" (OuterVolumeSpecName: "bundle") pod "b5808b89-67e6-4adf-886b-f3ff4499a1ff" (UID: "b5808b89-67e6-4adf-886b-f3ff4499a1ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:47:12.457365 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.457341 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-bundle" (OuterVolumeSpecName: "bundle") pod "2dc44f77-16a9-4713-b59d-c07093e1b1f0" (UID: "2dc44f77-16a9-4713-b59d-c07093e1b1f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:47:12.458801 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.458775 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5808b89-67e6-4adf-886b-f3ff4499a1ff-kube-api-access-fgmtk" (OuterVolumeSpecName: "kube-api-access-fgmtk") pod "b5808b89-67e6-4adf-886b-f3ff4499a1ff" (UID: "b5808b89-67e6-4adf-886b-f3ff4499a1ff"). InnerVolumeSpecName "kube-api-access-fgmtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:47:12.458964 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.458941 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc44f77-16a9-4713-b59d-c07093e1b1f0-kube-api-access-2c9dg" (OuterVolumeSpecName: "kube-api-access-2c9dg") pod "2dc44f77-16a9-4713-b59d-c07093e1b1f0" (UID: "2dc44f77-16a9-4713-b59d-c07093e1b1f0"). InnerVolumeSpecName "kube-api-access-2c9dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:47:12.462475 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.462454 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-util" (OuterVolumeSpecName: "util") pod "b5808b89-67e6-4adf-886b-f3ff4499a1ff" (UID: "b5808b89-67e6-4adf-886b-f3ff4499a1ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:47:12.462885 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.462866 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-util" (OuterVolumeSpecName: "util") pod "2dc44f77-16a9-4713-b59d-c07093e1b1f0" (UID: "2dc44f77-16a9-4713-b59d-c07093e1b1f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 20:47:12.557039 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.557018 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c9dg\" (UniqueName: \"kubernetes.io/projected/2dc44f77-16a9-4713-b59d-c07093e1b1f0-kube-api-access-2c9dg\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:12.557039 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.557039 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-util\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:12.557164 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.557049 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-util\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:12.557164 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.557057 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgmtk\" (UniqueName: \"kubernetes.io/projected/b5808b89-67e6-4adf-886b-f3ff4499a1ff-kube-api-access-fgmtk\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:12.557164 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.557067 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5808b89-67e6-4adf-886b-f3ff4499a1ff-bundle\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:12.557164 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.557076 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2dc44f77-16a9-4713-b59d-c07093e1b1f0-bundle\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:12.958486 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958458 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-9c4b4cfdd-njfxv"] Apr 16 20:47:12.958799 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958784 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53bbc7b6-a3e5-410e-9106-08828d972dee" containerName="util" Apr 16 20:47:12.958848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958800 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53bbc7b6-a3e5-410e-9106-08828d972dee" containerName="util" Apr 16 20:47:12.958848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958810 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5808b89-67e6-4adf-886b-f3ff4499a1ff" containerName="util" Apr 16 20:47:12.958848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958815 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5808b89-67e6-4adf-886b-f3ff4499a1ff" containerName="util" Apr 16 20:47:12.958848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958822 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53bbc7b6-a3e5-410e-9106-08828d972dee" containerName="pull" Apr 16 20:47:12.958848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958827 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53bbc7b6-a3e5-410e-9106-08828d972dee" containerName="pull" Apr 16 20:47:12.958848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958834 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53bbc7b6-a3e5-410e-9106-08828d972dee" containerName="extract" Apr 16 20:47:12.958848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958840 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53bbc7b6-a3e5-410e-9106-08828d972dee" containerName="extract" Apr 16 20:47:12.958848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958846 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5808b89-67e6-4adf-886b-f3ff4499a1ff" containerName="extract" Apr 16 20:47:12.958848 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958851 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5808b89-67e6-4adf-886b-f3ff4499a1ff" containerName="extract" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958861 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dace4755-9fcf-42d7-9ab4-e3a9d371f004" containerName="util" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958866 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dace4755-9fcf-42d7-9ab4-e3a9d371f004" containerName="util" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958872 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc44f77-16a9-4713-b59d-c07093e1b1f0" containerName="extract" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958877 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc44f77-16a9-4713-b59d-c07093e1b1f0" containerName="extract" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958886 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dace4755-9fcf-42d7-9ab4-e3a9d371f004" containerName="pull" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958891 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dace4755-9fcf-42d7-9ab4-e3a9d371f004" containerName="pull" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958899 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc44f77-16a9-4713-b59d-c07093e1b1f0" containerName="pull" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958904 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc44f77-16a9-4713-b59d-c07093e1b1f0" containerName="pull" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958910 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5808b89-67e6-4adf-886b-f3ff4499a1ff" containerName="pull" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958914 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5808b89-67e6-4adf-886b-f3ff4499a1ff" containerName="pull" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958921 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc44f77-16a9-4713-b59d-c07093e1b1f0" containerName="util" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958926 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc44f77-16a9-4713-b59d-c07093e1b1f0" containerName="util" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958934 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dace4755-9fcf-42d7-9ab4-e3a9d371f004" containerName="extract" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958939 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dace4755-9fcf-42d7-9ab4-e3a9d371f004" containerName="extract" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958978 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="53bbc7b6-a3e5-410e-9106-08828d972dee" containerName="extract" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958987 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b5808b89-67e6-4adf-886b-f3ff4499a1ff" containerName="extract" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.958994 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="dace4755-9fcf-42d7-9ab4-e3a9d371f004" containerName="extract" Apr 16 20:47:12.959089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.959000 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="2dc44f77-16a9-4713-b59d-c07093e1b1f0" containerName="extract" Apr 16 20:47:12.963398 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.963379 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:12.976464 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:12.976439 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9c4b4cfdd-njfxv"] Apr 16 20:47:13.061780 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.061746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-console-config\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.061885 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.061805 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-oauth-serving-cert\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.061921 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.061882 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12424dd7-aefc-476f-b59f-65b8a289c774-console-serving-cert\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.061921 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.061912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-service-ca\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.061993 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.061929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-trusted-ca-bundle\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.061993 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.061960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9hz\" (UniqueName: \"kubernetes.io/projected/12424dd7-aefc-476f-b59f-65b8a289c774-kube-api-access-gt9hz\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.061993 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.061978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12424dd7-aefc-476f-b59f-65b8a289c774-console-oauth-config\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.162569 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.162538 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-oauth-serving-cert\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.162708 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.162586 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12424dd7-aefc-476f-b59f-65b8a289c774-console-serving-cert\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.162708 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.162631 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-service-ca\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.162708 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.162654 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-trusted-ca-bundle\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.162887 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.162818 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9hz\" (UniqueName: \"kubernetes.io/projected/12424dd7-aefc-476f-b59f-65b8a289c774-kube-api-access-gt9hz\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.162887 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.162866 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12424dd7-aefc-476f-b59f-65b8a289c774-console-oauth-config\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.162990 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.162909 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-console-config\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.163370 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.163347 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-oauth-serving-cert\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.163463 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.163347 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-service-ca\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.163514 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.163461 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-trusted-ca-bundle\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.163548 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.163535 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12424dd7-aefc-476f-b59f-65b8a289c774-console-config\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.164984 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.164967 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12424dd7-aefc-476f-b59f-65b8a289c774-console-serving-cert\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.165185 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.165165 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12424dd7-aefc-476f-b59f-65b8a289c774-console-oauth-config\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.170886 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.170866 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9hz\" (UniqueName: \"kubernetes.io/projected/12424dd7-aefc-476f-b59f-65b8a289c774-kube-api-access-gt9hz\") pod \"console-9c4b4cfdd-njfxv\" (UID: \"12424dd7-aefc-476f-b59f-65b8a289c774\") " pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.262363 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.262305 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" event={"ID":"2dc44f77-16a9-4713-b59d-c07093e1b1f0","Type":"ContainerDied","Data":"537ac8cf09ccf254fdff50025dad948c1d426a82e2d57a2504b57b4c028c5e58"} Apr 16 20:47:13.262363 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.262336 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="537ac8cf09ccf254fdff50025dad948c1d426a82e2d57a2504b57b4c028c5e58" Apr 16 20:47:13.262363 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.262340 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm" Apr 16 20:47:13.264162 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.264140 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" event={"ID":"b5808b89-67e6-4adf-886b-f3ff4499a1ff","Type":"ContainerDied","Data":"57875dd8b98d91af00f5c7946deca805c2eacbc848d28739e2783a1cb4b1c88e"} Apr 16 20:47:13.264162 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.264168 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57875dd8b98d91af00f5c7946deca805c2eacbc848d28739e2783a1cb4b1c88e" Apr 16 20:47:13.264320 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.264187 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm" Apr 16 20:47:13.271952 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.271933 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:13.394464 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:13.394441 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9c4b4cfdd-njfxv"] Apr 16 20:47:13.396335 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:47:13.396304 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12424dd7_aefc_476f_b59f_65b8a289c774.slice/crio-0db97e47120d96d0253c73f5b74ed914ab1c1699f78947eb84a0e47ea7a93935 WatchSource:0}: Error finding container 0db97e47120d96d0253c73f5b74ed914ab1c1699f78947eb84a0e47ea7a93935: Status 404 returned error can't find the container with id 0db97e47120d96d0253c73f5b74ed914ab1c1699f78947eb84a0e47ea7a93935 Apr 16 20:47:14.269949 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:14.269916 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c4b4cfdd-njfxv" event={"ID":"12424dd7-aefc-476f-b59f-65b8a289c774","Type":"ContainerStarted","Data":"196b04d585c4a55c3c1793fc98e32496504b297abe71e700d4717d03747b9f80"} Apr 16 20:47:14.269949 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:14.269949 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c4b4cfdd-njfxv" event={"ID":"12424dd7-aefc-476f-b59f-65b8a289c774","Type":"ContainerStarted","Data":"0db97e47120d96d0253c73f5b74ed914ab1c1699f78947eb84a0e47ea7a93935"} Apr 16 20:47:14.288972 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:14.288929 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-9c4b4cfdd-njfxv" podStartSLOduration=2.28891716 podStartE2EDuration="2.28891716s" podCreationTimestamp="2026-04-16 20:47:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 20:47:14.287144406 +0000 UTC m=+564.462508673" watchObservedRunningTime="2026-04-16 20:47:14.28891716 +0000 UTC m=+564.464281425" Apr 16 20:47:22.277674 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:22.277638 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6"] Apr 16 20:47:22.281060 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:22.281041 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6" Apr 16 20:47:22.284254 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:22.284230 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Apr 16 20:47:22.284371 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:22.284253 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sv8sp\"" Apr 16 20:47:22.302421 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:22.302395 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6"] Apr 16 20:47:22.332550 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:22.332530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtj4\" (UniqueName: \"kubernetes.io/projected/03f33a24-6191-4f03-b0c4-200afd0eaee3-kube-api-access-nhtj4\") pod \"dns-operator-controller-manager-648d5c98bc-q8pd6\" (UID: \"03f33a24-6191-4f03-b0c4-200afd0eaee3\") " pod="kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6" Apr 16 20:47:22.433176 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:22.433148 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtj4\" (UniqueName: \"kubernetes.io/projected/03f33a24-6191-4f03-b0c4-200afd0eaee3-kube-api-access-nhtj4\") pod \"dns-operator-controller-manager-648d5c98bc-q8pd6\" (UID: \"03f33a24-6191-4f03-b0c4-200afd0eaee3\") " pod="kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6" Apr 16 20:47:22.448638 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:22.448590 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtj4\" (UniqueName: \"kubernetes.io/projected/03f33a24-6191-4f03-b0c4-200afd0eaee3-kube-api-access-nhtj4\") pod \"dns-operator-controller-manager-648d5c98bc-q8pd6\" (UID: \"03f33a24-6191-4f03-b0c4-200afd0eaee3\") " pod="kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6" Apr 16 20:47:22.591393 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:22.591366 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6" Apr 16 20:47:22.714089 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:22.714066 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6"] Apr 16 20:47:22.715971 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:47:22.715942 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f33a24_6191_4f03_b0c4_200afd0eaee3.slice/crio-10792dae961ac835cec10654fc566cf2e43b2c8d7a90a77417d9626f127ac292 WatchSource:0}: Error finding container 10792dae961ac835cec10654fc566cf2e43b2c8d7a90a77417d9626f127ac292: Status 404 returned error can't find the container with id 10792dae961ac835cec10654fc566cf2e43b2c8d7a90a77417d9626f127ac292 Apr 16 20:47:23.273114 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:23.273078 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:23.273282 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:23.273133 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:23.277956 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:23.277933 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:23.303509 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:23.303481 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6" event={"ID":"03f33a24-6191-4f03-b0c4-200afd0eaee3","Type":"ContainerStarted","Data":"10792dae961ac835cec10654fc566cf2e43b2c8d7a90a77417d9626f127ac292"} Apr 16 20:47:23.307294 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:23.307274 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9c4b4cfdd-njfxv" Apr 16 20:47:23.365870 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:23.365841 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bf85fbb7d-wth7c"] Apr 16 20:47:25.314453 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:25.314422 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6" event={"ID":"03f33a24-6191-4f03-b0c4-200afd0eaee3","Type":"ContainerStarted","Data":"0cd692b778302542756710a00ad9182d8ac9c62efc4837dd8ca20c2c4f247bec"} Apr 16 20:47:25.314803 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:25.314637 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6" Apr 16 20:47:25.331359 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:25.331318 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6" podStartSLOduration=0.925593191 podStartE2EDuration="3.331305496s" podCreationTimestamp="2026-04-16 20:47:22 +0000 UTC" firstStartedPulling="2026-04-16 20:47:22.718417141 +0000 UTC m=+572.893781384" lastFinishedPulling="2026-04-16 20:47:25.124129441 +0000 UTC m=+575.299493689" observedRunningTime="2026-04-16 20:47:25.329968479 +0000 UTC m=+575.505332746" watchObservedRunningTime="2026-04-16 20:47:25.331305496 +0000 UTC m=+575.506669762" Apr 16 20:47:29.549158 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:29.549109 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9"] Apr 16 20:47:29.555147 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:29.555127 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" Apr 16 20:47:29.558366 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:29.558343 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tcvmz\"" Apr 16 20:47:29.565374 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:29.565352 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9"] Apr 16 20:47:29.681620 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:29.681578 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gwch\" (UniqueName: \"kubernetes.io/projected/c8ce8b47-1043-46a3-b333-a8725942dc80-kube-api-access-5gwch\") pod \"limitador-operator-controller-manager-85c4996f8c-8xsj9\" (UID: \"c8ce8b47-1043-46a3-b333-a8725942dc80\") " pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" Apr 16 20:47:29.782517 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:29.782490 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwch\" (UniqueName: \"kubernetes.io/projected/c8ce8b47-1043-46a3-b333-a8725942dc80-kube-api-access-5gwch\") pod \"limitador-operator-controller-manager-85c4996f8c-8xsj9\" (UID: \"c8ce8b47-1043-46a3-b333-a8725942dc80\") " pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" Apr 16 20:47:29.795052 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:29.795028 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gwch\" (UniqueName: \"kubernetes.io/projected/c8ce8b47-1043-46a3-b333-a8725942dc80-kube-api-access-5gwch\") pod \"limitador-operator-controller-manager-85c4996f8c-8xsj9\" (UID: \"c8ce8b47-1043-46a3-b333-a8725942dc80\") " pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" Apr 16 20:47:29.865037 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:29.864987 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" Apr 16 20:47:29.985172 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:29.985028 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9"] Apr 16 20:47:29.987903 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:47:29.987876 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ce8b47_1043_46a3_b333_a8725942dc80.slice/crio-d980e738ac3f588ae11df094a6287e69da71300c40d115146637d97da64c93ed WatchSource:0}: Error finding container d980e738ac3f588ae11df094a6287e69da71300c40d115146637d97da64c93ed: Status 404 returned error can't find the container with id d980e738ac3f588ae11df094a6287e69da71300c40d115146637d97da64c93ed Apr 16 20:47:30.332765 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:30.332734 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" event={"ID":"c8ce8b47-1043-46a3-b333-a8725942dc80","Type":"ContainerStarted","Data":"d980e738ac3f588ae11df094a6287e69da71300c40d115146637d97da64c93ed"} Apr 16 20:47:32.341399 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:32.341365 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" event={"ID":"c8ce8b47-1043-46a3-b333-a8725942dc80","Type":"ContainerStarted","Data":"da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee"} Apr 16 20:47:32.341811 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:32.341466 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" Apr 16 20:47:32.369066 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:32.369020 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" podStartSLOduration=1.5183705760000001 podStartE2EDuration="3.369006505s" podCreationTimestamp="2026-04-16 20:47:29 +0000 UTC" firstStartedPulling="2026-04-16 20:47:29.989943657 +0000 UTC m=+580.165307902" lastFinishedPulling="2026-04-16 20:47:31.840579588 +0000 UTC m=+582.015943831" observedRunningTime="2026-04-16 20:47:32.366120752 +0000 UTC m=+582.541485021" watchObservedRunningTime="2026-04-16 20:47:32.369006505 +0000 UTC m=+582.544370771" Apr 16 20:47:36.320633 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:36.320585 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-648d5c98bc-q8pd6" Apr 16 20:47:42.674893 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.674860 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm"] Apr 16 20:47:42.679555 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.679536 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:42.682545 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.682519 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Apr 16 20:47:42.682545 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.682543 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c6jk2\"" Apr 16 20:47:42.682732 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.682545 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Apr 16 20:47:42.686886 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.686852 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm"] Apr 16 20:47:42.777010 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.776984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgz75\" (UniqueName: \"kubernetes.io/projected/ed555409-3a9b-43c0-ba01-594cf026ffe0-kube-api-access-qgz75\") pod \"kuadrant-console-plugin-6cb54b5c86-whgnm\" (UID: \"ed555409-3a9b-43c0-ba01-594cf026ffe0\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:42.777130 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.777025 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed555409-3a9b-43c0-ba01-594cf026ffe0-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-whgnm\" (UID: \"ed555409-3a9b-43c0-ba01-594cf026ffe0\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:42.777130 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.777087 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed555409-3a9b-43c0-ba01-594cf026ffe0-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-whgnm\" (UID: \"ed555409-3a9b-43c0-ba01-594cf026ffe0\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:42.877528 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.877501 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qgz75\" (UniqueName: \"kubernetes.io/projected/ed555409-3a9b-43c0-ba01-594cf026ffe0-kube-api-access-qgz75\") pod \"kuadrant-console-plugin-6cb54b5c86-whgnm\" (UID: \"ed555409-3a9b-43c0-ba01-594cf026ffe0\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:42.877661 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.877549 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed555409-3a9b-43c0-ba01-594cf026ffe0-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-whgnm\" (UID: \"ed555409-3a9b-43c0-ba01-594cf026ffe0\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:42.877661 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.877588 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed555409-3a9b-43c0-ba01-594cf026ffe0-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-whgnm\" (UID: \"ed555409-3a9b-43c0-ba01-594cf026ffe0\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:42.878163 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.878140 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed555409-3a9b-43c0-ba01-594cf026ffe0-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-whgnm\" (UID: \"ed555409-3a9b-43c0-ba01-594cf026ffe0\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:42.879973 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.879947 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed555409-3a9b-43c0-ba01-594cf026ffe0-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-whgnm\" (UID: \"ed555409-3a9b-43c0-ba01-594cf026ffe0\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:42.886058 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.886039 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgz75\" (UniqueName: \"kubernetes.io/projected/ed555409-3a9b-43c0-ba01-594cf026ffe0-kube-api-access-qgz75\") pod \"kuadrant-console-plugin-6cb54b5c86-whgnm\" (UID: \"ed555409-3a9b-43c0-ba01-594cf026ffe0\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:42.990026 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:42.989975 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" Apr 16 20:47:43.108731 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:43.108706 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm"] Apr 16 20:47:43.110241 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:47:43.110217 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded555409_3a9b_43c0_ba01_594cf026ffe0.slice/crio-c0178eb20b19ab5d777f64d95ee988882d5ee9ae289a9b2eff72b879464c7b38 WatchSource:0}: Error finding container c0178eb20b19ab5d777f64d95ee988882d5ee9ae289a9b2eff72b879464c7b38: Status 404 returned error can't find the container with id c0178eb20b19ab5d777f64d95ee988882d5ee9ae289a9b2eff72b879464c7b38 Apr 16 20:47:43.347155 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:43.347132 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" Apr 16 20:47:43.380066 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:43.380045 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" event={"ID":"ed555409-3a9b-43c0-ba01-594cf026ffe0","Type":"ContainerStarted","Data":"c0178eb20b19ab5d777f64d95ee988882d5ee9ae289a9b2eff72b879464c7b38"} Apr 16 20:47:48.389615 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.389537 2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-bf85fbb7d-wth7c" podUID="78c57ea5-7dbc-400c-94fa-ddd7f94288a1" containerName="console" containerID="cri-o://a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353" gracePeriod=15 Apr 16 20:47:48.652056 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.652034 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bf85fbb7d-wth7c_78c57ea5-7dbc-400c-94fa-ddd7f94288a1/console/0.log" Apr 16 20:47:48.652169 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.652096 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:47:48.729593 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.729560 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-config\") pod \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " Apr 16 20:47:48.729764 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.729633 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-serving-cert\") pod \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " Apr 16 20:47:48.729764 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.729662 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x92jr\" (UniqueName: \"kubernetes.io/projected/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-kube-api-access-x92jr\") pod \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " Apr 16 20:47:48.729764 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.729687 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-oauth-config\") pod \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " Apr 16 20:47:48.729764 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.729716 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-oauth-serving-cert\") pod \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " Apr 16 20:47:48.730205 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.730008 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-config" (OuterVolumeSpecName: "console-config") pod "78c57ea5-7dbc-400c-94fa-ddd7f94288a1" (UID: "78c57ea5-7dbc-400c-94fa-ddd7f94288a1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:47:48.730205 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.730022 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-trusted-ca-bundle\") pod \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " Apr 16 20:47:48.730205 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.730095 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-service-ca\") pod \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\" (UID: \"78c57ea5-7dbc-400c-94fa-ddd7f94288a1\") " Apr 16 20:47:48.730205 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.730137 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "78c57ea5-7dbc-400c-94fa-ddd7f94288a1" (UID: "78c57ea5-7dbc-400c-94fa-ddd7f94288a1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:47:48.730830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.730394 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "78c57ea5-7dbc-400c-94fa-ddd7f94288a1" (UID: "78c57ea5-7dbc-400c-94fa-ddd7f94288a1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:47:48.730830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.730645 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-service-ca" (OuterVolumeSpecName: "service-ca") pod "78c57ea5-7dbc-400c-94fa-ddd7f94288a1" (UID: "78c57ea5-7dbc-400c-94fa-ddd7f94288a1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 20:47:48.730830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.730752 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-oauth-serving-cert\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:48.730830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.730777 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-trusted-ca-bundle\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:48.730830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.730792 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-service-ca\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:48.730830 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.730807 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-config\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:48.732152 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.732128 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "78c57ea5-7dbc-400c-94fa-ddd7f94288a1" (UID: "78c57ea5-7dbc-400c-94fa-ddd7f94288a1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 20:47:48.732247 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.732210 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-kube-api-access-x92jr" (OuterVolumeSpecName: "kube-api-access-x92jr") pod "78c57ea5-7dbc-400c-94fa-ddd7f94288a1" (UID: "78c57ea5-7dbc-400c-94fa-ddd7f94288a1"). InnerVolumeSpecName "kube-api-access-x92jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:47:48.732440 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.732422 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "78c57ea5-7dbc-400c-94fa-ddd7f94288a1" (UID: "78c57ea5-7dbc-400c-94fa-ddd7f94288a1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 20:47:48.832061 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.832035 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-serving-cert\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:48.832061 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.832057 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x92jr\" (UniqueName: \"kubernetes.io/projected/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-kube-api-access-x92jr\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:48.832189 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:48.832067 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78c57ea5-7dbc-400c-94fa-ddd7f94288a1-console-oauth-config\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:47:49.402691 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:49.402660 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bf85fbb7d-wth7c_78c57ea5-7dbc-400c-94fa-ddd7f94288a1/console/0.log" Apr 16 20:47:49.403060 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:49.402698 2562 generic.go:358] "Generic (PLEG): container finished" podID="78c57ea5-7dbc-400c-94fa-ddd7f94288a1" containerID="a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353" exitCode=2 Apr 16 20:47:49.403060 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:49.402731 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf85fbb7d-wth7c" event={"ID":"78c57ea5-7dbc-400c-94fa-ddd7f94288a1","Type":"ContainerDied","Data":"a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353"} Apr 16 20:47:49.403060 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:49.402766 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf85fbb7d-wth7c" event={"ID":"78c57ea5-7dbc-400c-94fa-ddd7f94288a1","Type":"ContainerDied","Data":"a5bf33b244e9957c7b024821bfabc5a4f60533a4cd028eefe3bbaf7324f2e1ea"} Apr 16 20:47:49.403060 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:49.402766 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf85fbb7d-wth7c" Apr 16 20:47:49.403060 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:49.402780 2562 scope.go:117] "RemoveContainer" containerID="a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353" Apr 16 20:47:49.411362 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:49.411342 2562 scope.go:117] "RemoveContainer" containerID="a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353" Apr 16 20:47:49.411642 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:47:49.411619 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353\": container with ID starting with a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353 not found: ID does not exist" containerID="a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353" Apr 16 20:47:49.411767 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:49.411647 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353"} err="failed to get container status \"a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353\": rpc error: code = NotFound desc = could not find container \"a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353\": container with ID starting with a2bf8e0f6494b19afda4a10aa20861f61bc7c002b0e8fe4d7e9b387106114353 not found: ID does not exist" Apr 16 20:47:49.424823 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:49.424801 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bf85fbb7d-wth7c"] Apr 16 20:47:49.429421 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:49.429402 2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bf85fbb7d-wth7c"] Apr 16 20:47:50.445831 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:50.445796 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:47:50.446188 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:50.446076 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:47:50.503477 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:50.503448 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c57ea5-7dbc-400c-94fa-ddd7f94288a1" path="/var/lib/kubelet/pods/78c57ea5-7dbc-400c-94fa-ddd7f94288a1/volumes" Apr 16 20:47:52.361419 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.361389 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9"] Apr 16 20:47:52.361882 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.361691 2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" podUID="c8ce8b47-1043-46a3-b333-a8725942dc80" containerName="manager" containerID="cri-o://da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee" gracePeriod=2 Apr 16 20:47:52.376970 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.376940 2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9"] Apr 16 20:47:52.417190 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.417059 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4"] Apr 16 20:47:52.417515 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.417498 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c57ea5-7dbc-400c-94fa-ddd7f94288a1" containerName="console" Apr 16 20:47:52.417620 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.417517 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c57ea5-7dbc-400c-94fa-ddd7f94288a1" containerName="console" Apr 16 20:47:52.417620 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.417541 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ce8b47-1043-46a3-b333-a8725942dc80" containerName="manager" Apr 16 20:47:52.417620 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.417550 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ce8b47-1043-46a3-b333-a8725942dc80" containerName="manager" Apr 16 20:47:52.417783 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.417650 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="c8ce8b47-1043-46a3-b333-a8725942dc80" containerName="manager" Apr 16 20:47:52.417783 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.417666 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="78c57ea5-7dbc-400c-94fa-ddd7f94288a1" containerName="console" Apr 16 20:47:52.420739 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.420716 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4" Apr 16 20:47:52.424269 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.424242 2562 status_manager.go:895] "Failed to get status for pod" podUID="c8ce8b47-1043-46a3-b333-a8725942dc80" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" err="pods \"limitador-operator-controller-manager-85c4996f8c-8xsj9\" is forbidden: User \"system:node:ip-10-0-142-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-90.ec2.internal' and this object" Apr 16 20:47:52.434440 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.434083 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4"] Apr 16 20:47:52.560515 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.560478 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drwcd\" (UniqueName: \"kubernetes.io/projected/8fcfe231-6a16-491b-a4eb-f144cadc9fbe-kube-api-access-drwcd\") pod \"limitador-operator-controller-manager-85c4996f8c-2h5w4\" (UID: \"8fcfe231-6a16-491b-a4eb-f144cadc9fbe\") " pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4" Apr 16 20:47:52.661971 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.661883 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-drwcd\" (UniqueName: \"kubernetes.io/projected/8fcfe231-6a16-491b-a4eb-f144cadc9fbe-kube-api-access-drwcd\") pod \"limitador-operator-controller-manager-85c4996f8c-2h5w4\" (UID: \"8fcfe231-6a16-491b-a4eb-f144cadc9fbe\") " pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4" Apr 16 20:47:52.675271 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.675239 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drwcd\" (UniqueName: \"kubernetes.io/projected/8fcfe231-6a16-491b-a4eb-f144cadc9fbe-kube-api-access-drwcd\") pod \"limitador-operator-controller-manager-85c4996f8c-2h5w4\" (UID: \"8fcfe231-6a16-491b-a4eb-f144cadc9fbe\") " pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4" Apr 16 20:47:52.741522 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:47:52.741486 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4" Apr 16 20:48:05.879792 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:05.879768 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" Apr 16 20:48:05.882405 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:05.882377 2562 status_manager.go:895] "Failed to get status for pod" podUID="c8ce8b47-1043-46a3-b333-a8725942dc80" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" err="pods \"limitador-operator-controller-manager-85c4996f8c-8xsj9\" is forbidden: User \"system:node:ip-10-0-142-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-90.ec2.internal' and this object" Apr 16 20:48:05.911358 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:05.911334 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4"] Apr 16 20:48:05.914230 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:48:05.914201 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcfe231_6a16_491b_a4eb_f144cadc9fbe.slice/crio-77cb36879fcdb075e66b9d70d82bdb14fe329fc36bbad7c7f036c1fcb8554dc8 WatchSource:0}: Error finding container 77cb36879fcdb075e66b9d70d82bdb14fe329fc36bbad7c7f036c1fcb8554dc8: Status 404 returned error can't find the container with id 77cb36879fcdb075e66b9d70d82bdb14fe329fc36bbad7c7f036c1fcb8554dc8 Apr 16 20:48:05.984703 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:05.984677 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gwch\" (UniqueName: \"kubernetes.io/projected/c8ce8b47-1043-46a3-b333-a8725942dc80-kube-api-access-5gwch\") pod \"c8ce8b47-1043-46a3-b333-a8725942dc80\" (UID: \"c8ce8b47-1043-46a3-b333-a8725942dc80\") " Apr 16 20:48:05.986406 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:05.986384 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ce8b47-1043-46a3-b333-a8725942dc80-kube-api-access-5gwch" (OuterVolumeSpecName: "kube-api-access-5gwch") pod "c8ce8b47-1043-46a3-b333-a8725942dc80" (UID: "c8ce8b47-1043-46a3-b333-a8725942dc80"). InnerVolumeSpecName "kube-api-access-5gwch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 20:48:06.085650 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.085622 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gwch\" (UniqueName: \"kubernetes.io/projected/c8ce8b47-1043-46a3-b333-a8725942dc80-kube-api-access-5gwch\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 20:48:06.471001 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.470913 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4" event={"ID":"8fcfe231-6a16-491b-a4eb-f144cadc9fbe","Type":"ContainerStarted","Data":"54bf8ddc91f72aff4c26d6ea00ad57fea7adbba83f119cda6527c89bfdefced2"} Apr 16 20:48:06.471001 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.470956 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4" event={"ID":"8fcfe231-6a16-491b-a4eb-f144cadc9fbe","Type":"ContainerStarted","Data":"77cb36879fcdb075e66b9d70d82bdb14fe329fc36bbad7c7f036c1fcb8554dc8"} Apr 16 20:48:06.471001 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.470998 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4" Apr 16 20:48:06.472221 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.472193 2562 generic.go:358] "Generic (PLEG): container finished" podID="c8ce8b47-1043-46a3-b333-a8725942dc80" containerID="da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee" exitCode=0 Apr 16 20:48:06.472335 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.472239 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" Apr 16 20:48:06.472335 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.472290 2562 scope.go:117] "RemoveContainer" containerID="da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee" Apr 16 20:48:06.473747 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.473719 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" event={"ID":"ed555409-3a9b-43c0-ba01-594cf026ffe0","Type":"ContainerStarted","Data":"ff67b9b81410dcb4569cdf500d5384e4aea8686c6ca6e1a3fe5ce2b73d551969"} Apr 16 20:48:06.476945 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.476916 2562 status_manager.go:895] "Failed to get status for pod" podUID="c8ce8b47-1043-46a3-b333-a8725942dc80" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-8xsj9" err="pods \"limitador-operator-controller-manager-85c4996f8c-8xsj9\" is forbidden: User \"system:node:ip-10-0-142-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-90.ec2.internal' and this object" Apr 16 20:48:06.481797 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.481776 2562 scope.go:117] "RemoveContainer" containerID="da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee" Apr 16 20:48:06.482109 ip-10-0-142-90 kubenswrapper[2562]: E0416 20:48:06.482089 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee\": container with ID starting with da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee not found: ID does not exist" containerID="da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee" Apr 16 20:48:06.482186 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.482117 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee"} err="failed to get container status \"da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee\": rpc error: code = NotFound desc = could not find container \"da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee\": container with ID starting with da8e360250f23b48061d4046d2f333990c5093d176b489e04cd4ee4f196cbbee not found: ID does not exist" Apr 16 20:48:06.501821 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.501794 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ce8b47-1043-46a3-b333-a8725942dc80" path="/var/lib/kubelet/pods/c8ce8b47-1043-46a3-b333-a8725942dc80/volumes" Apr 16 20:48:06.509305 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.509265 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4" podStartSLOduration=14.509253552 podStartE2EDuration="14.509253552s" podCreationTimestamp="2026-04-16 20:47:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 20:48:06.50764635 +0000 UTC m=+616.683010619" watchObservedRunningTime="2026-04-16 20:48:06.509253552 +0000 UTC m=+616.684617818" Apr 16 20:48:06.539977 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:06.539941 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-whgnm" podStartSLOduration=1.813218205 podStartE2EDuration="24.539931022s" podCreationTimestamp="2026-04-16 20:47:42 +0000 UTC" firstStartedPulling="2026-04-16 20:47:43.111482845 +0000 UTC m=+593.286847089" lastFinishedPulling="2026-04-16 20:48:05.838195643 +0000 UTC m=+616.013559906" observedRunningTime="2026-04-16 20:48:06.538214096 +0000 UTC m=+616.713578361" watchObservedRunningTime="2026-04-16 20:48:06.539931022 +0000 UTC m=+616.715295287" Apr 16 20:48:17.480947 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:17.480911 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-2h5w4" Apr 16 20:48:21.848580 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:21.848547 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29"] Apr 16 20:48:21.867325 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:21.867303 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29"] Apr 16 20:48:21.867446 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:21.867413 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:21.870109 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:21.870088 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-z65k2\"" Apr 16 20:48:21.999641 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:21.999596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-workload-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:21.999756 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:21.999654 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:21.999756 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:21.999695 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fcb52d-f173-4e19-8a69-90a678a0164e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.004623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.000140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrfg6\" (UniqueName: \"kubernetes.io/projected/b8fcb52d-f173-4e19-8a69-90a678a0164e-kube-api-access-xrfg6\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.004623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.000308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-workload-certs\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.004623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.000346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-token\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.004623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.000396 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-credential-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.004623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.000436 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.004623 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.000472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-data\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101301 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101243 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101301 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101276 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fcb52d-f173-4e19-8a69-90a678a0164e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101452 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101304 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xrfg6\" (UniqueName: \"kubernetes.io/projected/b8fcb52d-f173-4e19-8a69-90a678a0164e-kube-api-access-xrfg6\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101452 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101335 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-workload-certs\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101452 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101363 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-token\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101452 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101401 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-credential-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101452 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101427 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101742 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101637 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-data\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101742 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101715 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-workload-certs\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101742 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101734 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-workload-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101907 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101833 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-credential-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.101993 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.101972 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-data\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.102051 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.102029 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-workload-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.102107 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.102084 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fcb52d-f173-4e19-8a69-90a678a0164e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.103571 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.103551 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.103893 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.103872 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.112273 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.112251 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fcb52d-f173-4e19-8a69-90a678a0164e-istio-token\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.112797 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.112778 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrfg6\" (UniqueName: \"kubernetes.io/projected/b8fcb52d-f173-4e19-8a69-90a678a0164e-kube-api-access-xrfg6\") pod \"maas-default-gateway-openshift-default-58b6f876-jvj29\" (UID: \"b8fcb52d-f173-4e19-8a69-90a678a0164e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.180745 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.180718 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:22.323744 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.323718 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29"] Apr 16 20:48:22.325889 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:48:22.325860 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fcb52d_f173_4e19_8a69_90a678a0164e.slice/crio-51c357a9185292b45903eb8234fb79ad0ea81c3c2c3f747b9c2e2c498161decf WatchSource:0}: Error finding container 51c357a9185292b45903eb8234fb79ad0ea81c3c2c3f747b9c2e2c498161decf: Status 404 returned error can't find the container with id 51c357a9185292b45903eb8234fb79ad0ea81c3c2c3f747b9c2e2c498161decf Apr 16 20:48:22.327703 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.327683 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 20:48:22.328075 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.328043 2562 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Apr 16 20:48:22.328145 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.328102 2562 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Apr 16 20:48:22.328145 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.328131 2562 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Apr 16 20:48:22.534202 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.534165 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" event={"ID":"b8fcb52d-f173-4e19-8a69-90a678a0164e","Type":"ContainerStarted","Data":"b5dea71e406e0dffd5e97e30cf071ad870f0356eacc8586e22c9b0e5820e7249"} Apr 16 20:48:22.534202 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.534203 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" event={"ID":"b8fcb52d-f173-4e19-8a69-90a678a0164e","Type":"ContainerStarted","Data":"51c357a9185292b45903eb8234fb79ad0ea81c3c2c3f747b9c2e2c498161decf"} Apr 16 20:48:22.554232 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:22.554180 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" podStartSLOduration=1.554166004 podStartE2EDuration="1.554166004s" podCreationTimestamp="2026-04-16 20:48:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 20:48:22.552869247 +0000 UTC m=+632.728233514" watchObservedRunningTime="2026-04-16 20:48:22.554166004 +0000 UTC m=+632.729530269" Apr 16 20:48:23.181583 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:23.181548 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:23.186082 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:23.186059 2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:23.538696 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:23.538620 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:23.539587 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:23.539568 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jvj29" Apr 16 20:48:26.995006 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:26.994974 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:48:27.061989 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.061955 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:48:27.062155 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.062067 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" Apr 16 20:48:27.063387 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.063360 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:48:27.065942 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.065911 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Apr 16 20:48:27.136562 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.136538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/001cc2c6-4ef2-4879-8a91-ed2c6e58bc63-config-file\") pod \"limitador-limitador-78c99df468-bz4sz\" (UID: \"001cc2c6-4ef2-4879-8a91-ed2c6e58bc63\") " pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" Apr 16 20:48:27.136667 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.136580 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclv4\" (UniqueName: \"kubernetes.io/projected/001cc2c6-4ef2-4879-8a91-ed2c6e58bc63-kube-api-access-sclv4\") pod \"limitador-limitador-78c99df468-bz4sz\" (UID: \"001cc2c6-4ef2-4879-8a91-ed2c6e58bc63\") " pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" Apr 16 20:48:27.237726 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.237699 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sclv4\" (UniqueName: \"kubernetes.io/projected/001cc2c6-4ef2-4879-8a91-ed2c6e58bc63-kube-api-access-sclv4\") pod \"limitador-limitador-78c99df468-bz4sz\" (UID: \"001cc2c6-4ef2-4879-8a91-ed2c6e58bc63\") " pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" Apr 16 20:48:27.237831 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.237799 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/001cc2c6-4ef2-4879-8a91-ed2c6e58bc63-config-file\") pod \"limitador-limitador-78c99df468-bz4sz\" (UID: \"001cc2c6-4ef2-4879-8a91-ed2c6e58bc63\") " pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" Apr 16 20:48:27.238348 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.238326 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/001cc2c6-4ef2-4879-8a91-ed2c6e58bc63-config-file\") pod \"limitador-limitador-78c99df468-bz4sz\" (UID: \"001cc2c6-4ef2-4879-8a91-ed2c6e58bc63\") " pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" Apr 16 20:48:27.259108 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.259056 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclv4\" (UniqueName: \"kubernetes.io/projected/001cc2c6-4ef2-4879-8a91-ed2c6e58bc63-kube-api-access-sclv4\") pod \"limitador-limitador-78c99df468-bz4sz\" (UID: \"001cc2c6-4ef2-4879-8a91-ed2c6e58bc63\") " pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" Apr 16 20:48:27.372365 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.372337 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" Apr 16 20:48:27.707209 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:27.707182 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:48:27.708935 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:48:27.708902 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001cc2c6_4ef2_4879_8a91_ed2c6e58bc63.slice/crio-0d2b97278fd1b139400eaff5f3c91947e72723616635b8ba3b3e4a2b54837fd5 WatchSource:0}: Error finding container 0d2b97278fd1b139400eaff5f3c91947e72723616635b8ba3b3e4a2b54837fd5: Status 404 returned error can't find the container with id 0d2b97278fd1b139400eaff5f3c91947e72723616635b8ba3b3e4a2b54837fd5 Apr 16 20:48:28.557867 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:28.557816 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" event={"ID":"001cc2c6-4ef2-4879-8a91-ed2c6e58bc63","Type":"ContainerStarted","Data":"0d2b97278fd1b139400eaff5f3c91947e72723616635b8ba3b3e4a2b54837fd5"} Apr 16 20:48:31.569619 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:31.569577 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" event={"ID":"001cc2c6-4ef2-4879-8a91-ed2c6e58bc63","Type":"ContainerStarted","Data":"07a3d9b1f56e0b36e434acc375d86424a99b0338b232f7295f3347706f90035d"} Apr 16 20:48:31.569984 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:31.569739 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" Apr 16 20:48:31.587718 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:31.587675 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" podStartSLOduration=2.519790558 podStartE2EDuration="5.587662224s" podCreationTimestamp="2026-04-16 20:48:26 +0000 UTC" firstStartedPulling="2026-04-16 20:48:27.710923318 +0000 UTC m=+637.886287579" lastFinishedPulling="2026-04-16 20:48:30.778795002 +0000 UTC m=+640.954159245" observedRunningTime="2026-04-16 20:48:31.585100921 +0000 UTC m=+641.760465187" watchObservedRunningTime="2026-04-16 20:48:31.587662224 +0000 UTC m=+641.763026499" Apr 16 20:48:42.574393 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:48:42.574366 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-78c99df468-bz4sz" Apr 16 20:49:01.446655 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:49:01.446622 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:49:44.063432 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:49:44.063355 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:49:52.327322 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:49:52.327288 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:49:56.924354 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:49:56.924323 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:50:08.028048 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:50:08.028012 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:50:33.737750 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:50:33.737715 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:51:01.530009 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:51:01.529967 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:51:48.632871 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:51:48.632838 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:51:58.438739 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:51:58.438706 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:52:06.738429 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:06.738385 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:52:18.426682 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:18.426646 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:52:25.931671 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:25.931635 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:52:37.435706 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:37.435670 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:52:45.256683 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:45.256591 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-764b68564c-75h4v"] Apr 16 20:52:45.258916 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:45.258895 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764b68564c-75h4v" Apr 16 20:52:45.261741 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:45.261722 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-c92v8\"" Apr 16 20:52:45.267553 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:45.267530 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764b68564c-75h4v"] Apr 16 20:52:45.381878 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:45.381852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9nxc\" (UniqueName: \"kubernetes.io/projected/0907960e-97cf-48f2-a256-f59d92f13615-kube-api-access-d9nxc\") pod \"maas-controller-764b68564c-75h4v\" (UID: \"0907960e-97cf-48f2-a256-f59d92f13615\") " pod="opendatahub/maas-controller-764b68564c-75h4v" Apr 16 20:52:45.482961 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:45.482939 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d9nxc\" (UniqueName: \"kubernetes.io/projected/0907960e-97cf-48f2-a256-f59d92f13615-kube-api-access-d9nxc\") pod \"maas-controller-764b68564c-75h4v\" (UID: \"0907960e-97cf-48f2-a256-f59d92f13615\") " pod="opendatahub/maas-controller-764b68564c-75h4v" Apr 16 20:52:45.491863 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:45.491841 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9nxc\" (UniqueName: \"kubernetes.io/projected/0907960e-97cf-48f2-a256-f59d92f13615-kube-api-access-d9nxc\") pod \"maas-controller-764b68564c-75h4v\" (UID: \"0907960e-97cf-48f2-a256-f59d92f13615\") " pod="opendatahub/maas-controller-764b68564c-75h4v" Apr 16 20:52:45.587063 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:45.587041 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764b68564c-75h4v" Apr 16 20:52:45.906572 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:45.906546 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764b68564c-75h4v"] Apr 16 20:52:45.908568 ip-10-0-142-90 kubenswrapper[2562]: W0416 20:52:45.908544 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0907960e_97cf_48f2_a256_f59d92f13615.slice/crio-598ee0c518217e0b41631d853d2b2b1d67029d45633e9bdc5d10e72520bc25f1 WatchSource:0}: Error finding container 598ee0c518217e0b41631d853d2b2b1d67029d45633e9bdc5d10e72520bc25f1: Status 404 returned error can't find the container with id 598ee0c518217e0b41631d853d2b2b1d67029d45633e9bdc5d10e72520bc25f1 Apr 16 20:52:46.481950 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:46.481899 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764b68564c-75h4v" event={"ID":"0907960e-97cf-48f2-a256-f59d92f13615","Type":"ContainerStarted","Data":"598ee0c518217e0b41631d853d2b2b1d67029d45633e9bdc5d10e72520bc25f1"} Apr 16 20:52:48.491796 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:48.491700 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764b68564c-75h4v" event={"ID":"0907960e-97cf-48f2-a256-f59d92f13615","Type":"ContainerStarted","Data":"c9887e17c6d2053831e397c2d391e0a62cc33527f4d35b821936c1f66f2a43e0"} Apr 16 20:52:48.491796 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:48.491780 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-764b68564c-75h4v" Apr 16 20:52:48.518766 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:48.518715 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/maas-controller-764b68564c-75h4v" podStartSLOduration=1.279530341 podStartE2EDuration="3.518701189s" podCreationTimestamp="2026-04-16 20:52:45 +0000 UTC" firstStartedPulling="2026-04-16 20:52:45.909770734 +0000 UTC m=+896.085134978" lastFinishedPulling="2026-04-16 20:52:48.148941582 +0000 UTC m=+898.324305826" observedRunningTime="2026-04-16 20:52:48.517408555 +0000 UTC m=+898.692772821" watchObservedRunningTime="2026-04-16 20:52:48.518701189 +0000 UTC m=+898.694065454" Apr 16 20:52:50.480484 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:50.480461 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:52:50.480484 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:50.480473 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:52:59.501691 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:52:59.501661 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-764b68564c-75h4v" Apr 16 20:53:41.436234 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:53:41.436198 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:53:56.224742 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:53:56.224708 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:54:35.532184 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:54:35.532152 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:54:52.826236 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:54:52.826200 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:55:06.237332 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:55:06.237292 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:55:22.525391 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:55:22.525357 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:55:26.225825 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:55:26.225792 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:56:13.724641 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:56:13.724592 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:56:22.730039 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:56:22.730002 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:56:40.270651 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:56:40.270616 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:56:49.032386 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:56:49.032344 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:57:05.934122 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:57:05.934087 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:57:14.234462 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:57:14.234429 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:57:46.732595 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:57:46.732556 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:57:50.514464 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:57:50.514436 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:57:50.516242 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:57:50.516220 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 20:57:54.530964 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:57:54.530932 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:58:03.926459 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:58:03.926424 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:58:12.127417 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:58:12.127382 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:58:20.033909 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:58:20.033875 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:58:37.026717 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:58:37.026686 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:58:47.629484 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:58:47.629450 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:59:35.124070 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:59:35.124030 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:59:43.228880 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:59:43.228850 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 20:59:53.128012 ip-10-0-142-90 kubenswrapper[2562]: I0416 20:59:53.127974 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:00:00.147793 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:00.147758 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29606220-wvxvg"] Apr 16 21:00:00.151204 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:00.151180 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" Apr 16 21:00:00.153978 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:00.153955 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-nj46q\"" Apr 16 21:00:00.170897 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:00.170875 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29606220-wvxvg"] Apr 16 21:00:00.274750 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:00.274717 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswqv\" (UniqueName: \"kubernetes.io/projected/db048725-ff01-489c-8272-c4057dc83b88-kube-api-access-qswqv\") pod \"maas-api-key-cleanup-29606220-wvxvg\" (UID: \"db048725-ff01-489c-8272-c4057dc83b88\") " pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" Apr 16 21:00:00.376171 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:00.376127 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qswqv\" (UniqueName: \"kubernetes.io/projected/db048725-ff01-489c-8272-c4057dc83b88-kube-api-access-qswqv\") pod \"maas-api-key-cleanup-29606220-wvxvg\" (UID: \"db048725-ff01-489c-8272-c4057dc83b88\") " pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" Apr 16 21:00:00.384877 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:00.384848 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswqv\" (UniqueName: \"kubernetes.io/projected/db048725-ff01-489c-8272-c4057dc83b88-kube-api-access-qswqv\") pod \"maas-api-key-cleanup-29606220-wvxvg\" (UID: \"db048725-ff01-489c-8272-c4057dc83b88\") " pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" Apr 16 21:00:00.462954 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:00.462893 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" Apr 16 21:00:00.792184 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:00.792160 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29606220-wvxvg"] Apr 16 21:00:00.793887 ip-10-0-142-90 kubenswrapper[2562]: W0416 21:00:00.793862 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb048725_ff01_489c_8272_c4057dc83b88.slice/crio-94234065e74ead8a9a38562bfd229d536ab65b3fcabf5398f3320fb68d0073db WatchSource:0}: Error finding container 94234065e74ead8a9a38562bfd229d536ab65b3fcabf5398f3320fb68d0073db: Status 404 returned error can't find the container with id 94234065e74ead8a9a38562bfd229d536ab65b3fcabf5398f3320fb68d0073db Apr 16 21:00:00.795487 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:00.795472 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 21:00:01.022217 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:01.022181 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" event={"ID":"db048725-ff01-489c-8272-c4057dc83b88","Type":"ContainerStarted","Data":"94234065e74ead8a9a38562bfd229d536ab65b3fcabf5398f3320fb68d0073db"} Apr 16 21:00:01.633478 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:01.633446 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:00:04.034625 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:04.034527 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" event={"ID":"db048725-ff01-489c-8272-c4057dc83b88","Type":"ContainerStarted","Data":"14b05f6b4434cb2892f8f6c422aa0b58621e993ee50ce182179e75a2ab58428e"} Apr 16 21:00:04.051443 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:04.051395 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" podStartSLOduration=1.579946218 podStartE2EDuration="4.051382818s" podCreationTimestamp="2026-04-16 21:00:00 +0000 UTC" firstStartedPulling="2026-04-16 21:00:00.795595045 +0000 UTC m=+1330.970959288" lastFinishedPulling="2026-04-16 21:00:03.267031629 +0000 UTC m=+1333.442395888" observedRunningTime="2026-04-16 21:00:04.051116272 +0000 UTC m=+1334.226480538" watchObservedRunningTime="2026-04-16 21:00:04.051382818 +0000 UTC m=+1334.226747084" Apr 16 21:00:10.836445 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:10.836411 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:00:19.731367 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:19.731330 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:00:24.110402 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:24.110322 2562 generic.go:358] "Generic (PLEG): container finished" podID="db048725-ff01-489c-8272-c4057dc83b88" containerID="14b05f6b4434cb2892f8f6c422aa0b58621e993ee50ce182179e75a2ab58428e" exitCode=6 Apr 16 21:00:24.110737 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:24.110406 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" event={"ID":"db048725-ff01-489c-8272-c4057dc83b88","Type":"ContainerDied","Data":"14b05f6b4434cb2892f8f6c422aa0b58621e993ee50ce182179e75a2ab58428e"} Apr 16 21:00:24.110737 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:24.110730 2562 scope.go:117] "RemoveContainer" containerID="14b05f6b4434cb2892f8f6c422aa0b58621e993ee50ce182179e75a2ab58428e" Apr 16 21:00:25.115332 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:25.115296 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" event={"ID":"db048725-ff01-489c-8272-c4057dc83b88","Type":"ContainerStarted","Data":"ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b"} Apr 16 21:00:28.236620 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:28.236582 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:00:36.549588 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:36.549549 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:00:45.187829 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:45.187802 2562 generic.go:358] "Generic (PLEG): container finished" podID="db048725-ff01-489c-8272-c4057dc83b88" containerID="ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b" exitCode=6 Apr 16 21:00:45.188218 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:45.187866 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" event={"ID":"db048725-ff01-489c-8272-c4057dc83b88","Type":"ContainerDied","Data":"ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b"} Apr 16 21:00:45.188218 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:45.187902 2562 scope.go:117] "RemoveContainer" containerID="14b05f6b4434cb2892f8f6c422aa0b58621e993ee50ce182179e75a2ab58428e" Apr 16 21:00:45.188317 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:45.188230 2562 scope.go:117] "RemoveContainer" containerID="ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b" Apr 16 21:00:45.188459 ip-10-0-142-90 kubenswrapper[2562]: E0416 21:00:45.188438 2562 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29606220-wvxvg_opendatahub(db048725-ff01-489c-8272-c4057dc83b88)\"" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" podUID="db048725-ff01-489c-8272-c4057dc83b88" Apr 16 21:00:45.340847 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:45.340816 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:00:54.867995 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:54.867964 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:00:58.498095 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:58.498053 2562 scope.go:117] "RemoveContainer" containerID="ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b" Apr 16 21:00:59.244487 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:59.244453 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" event={"ID":"db048725-ff01-489c-8272-c4057dc83b88","Type":"ContainerStarted","Data":"cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5"} Apr 16 21:00:59.538046 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:00:59.537967 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-29606220-wvxvg"] Apr 16 21:01:00.249186 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:00.249138 2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" podUID="db048725-ff01-489c-8272-c4057dc83b88" containerName="cleanup" containerID="cri-o://cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5" gracePeriod=30 Apr 16 21:01:03.251465 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:03.251432 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:01:12.154674 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:12.154638 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:01:19.286967 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.286945 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" Apr 16 21:01:19.317164 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.317098 2562 generic.go:358] "Generic (PLEG): container finished" podID="db048725-ff01-489c-8272-c4057dc83b88" containerID="cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5" exitCode=6 Apr 16 21:01:19.317258 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.317159 2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" Apr 16 21:01:19.317258 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.317193 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" event={"ID":"db048725-ff01-489c-8272-c4057dc83b88","Type":"ContainerDied","Data":"cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5"} Apr 16 21:01:19.317258 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.317229 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29606220-wvxvg" event={"ID":"db048725-ff01-489c-8272-c4057dc83b88","Type":"ContainerDied","Data":"94234065e74ead8a9a38562bfd229d536ab65b3fcabf5398f3320fb68d0073db"} Apr 16 21:01:19.317258 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.317251 2562 scope.go:117] "RemoveContainer" containerID="cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5" Apr 16 21:01:19.325817 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.325792 2562 scope.go:117] "RemoveContainer" containerID="ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b" Apr 16 21:01:19.333625 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.333586 2562 scope.go:117] "RemoveContainer" containerID="cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5" Apr 16 21:01:19.333966 ip-10-0-142-90 kubenswrapper[2562]: E0416 21:01:19.333946 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5\": container with ID starting with cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5 not found: ID does not exist" containerID="cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5" Apr 16 21:01:19.334039 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.333976 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5"} err="failed to get container status \"cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5\": rpc error: code = NotFound desc = could not find container \"cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5\": container with ID starting with cffcba1504a216df6c426dc7df8f5f46b32314f9c6e0f340938521c2e42374a5 not found: ID does not exist" Apr 16 21:01:19.334039 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.333998 2562 scope.go:117] "RemoveContainer" containerID="ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b" Apr 16 21:01:19.334258 ip-10-0-142-90 kubenswrapper[2562]: E0416 21:01:19.334235 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b\": container with ID starting with ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b not found: ID does not exist" containerID="ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b" Apr 16 21:01:19.334306 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.334269 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b"} err="failed to get container status \"ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b\": rpc error: code = NotFound desc = could not find container \"ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b\": container with ID starting with ba8f1789f7d1b0ba1e4468735eafe2a4da845ebb70cd3a71975dc0a50edb342b not found: ID does not exist" Apr 16 21:01:19.362517 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.362490 2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qswqv\" (UniqueName: \"kubernetes.io/projected/db048725-ff01-489c-8272-c4057dc83b88-kube-api-access-qswqv\") pod \"db048725-ff01-489c-8272-c4057dc83b88\" (UID: \"db048725-ff01-489c-8272-c4057dc83b88\") " Apr 16 21:01:19.364538 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.364504 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db048725-ff01-489c-8272-c4057dc83b88-kube-api-access-qswqv" (OuterVolumeSpecName: "kube-api-access-qswqv") pod "db048725-ff01-489c-8272-c4057dc83b88" (UID: "db048725-ff01-489c-8272-c4057dc83b88"). InnerVolumeSpecName "kube-api-access-qswqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 21:01:19.463680 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.463651 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qswqv\" (UniqueName: \"kubernetes.io/projected/db048725-ff01-489c-8272-c4057dc83b88-kube-api-access-qswqv\") on node \"ip-10-0-142-90.ec2.internal\" DevicePath \"\"" Apr 16 21:01:19.648335 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.648314 2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-29606220-wvxvg"] Apr 16 21:01:19.660737 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:19.660717 2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-29606220-wvxvg"] Apr 16 21:01:20.507085 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:20.507052 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db048725-ff01-489c-8272-c4057dc83b88" path="/var/lib/kubelet/pods/db048725-ff01-489c-8272-c4057dc83b88/volumes" Apr 16 21:01:22.261324 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:22.261293 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:01:29.424106 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:29.424058 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:01:38.608244 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:38.608213 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:01:46.808849 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:46.808774 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:01:56.343516 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:01:56.343471 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:02:04.768951 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:02:04.768913 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:02:50.541380 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:02:50.541353 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 21:02:50.544398 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:02:50.544374 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 21:04:21.436144 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:04:21.436105 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:04:28.434300 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:04:28.434267 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:04:52.736221 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:04:52.736147 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:04:57.734143 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:04:57.734106 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:05:07.330847 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:05:07.330814 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:05:17.838412 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:05:17.838374 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:05:26.929668 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:05:26.929631 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:05:37.035726 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:05:37.035692 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:05:45.631942 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:05:45.631908 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:05:56.453166 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:05:56.453131 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:06:06.436927 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:06:06.436893 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:06:16.232312 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:06:16.232277 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:06:24.636423 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:06:24.636389 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:06:57.634849 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:06:57.634806 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:07:41.129484 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:07:41.129406 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:07:48.949996 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:07:48.949960 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:07:50.577775 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:07:50.577749 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 21:07:50.579789 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:07:50.579767 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 21:07:58.537133 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:07:58.537088 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:08:06.435164 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:08:06.435124 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:08:15.550665 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:08:15.550633 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:08:28.137441 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:08:28.137405 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:08:36.735560 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:08:36.735524 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:08:44.447065 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:08:44.447024 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:08:53.344702 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:08:53.344668 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:09:02.149894 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:09:02.149858 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:09:10.030154 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:09:10.030072 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:09:21.328716 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:09:21.328675 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:09:38.543528 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:09:38.543494 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:09:46.431880 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:09:46.431845 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:09:55.030743 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:09:55.030703 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:10:03.731899 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:10:03.731864 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:10:20.943683 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:10:20.943648 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:10:29.173205 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:10:29.173162 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:10:38.629637 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:10:38.629589 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:10:46.529163 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:10:46.529079 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:10:56.337940 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:10:56.337896 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:11:04.127789 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:11:04.127751 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:11:12.732899 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:11:12.732862 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:11:23.927368 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:11:23.927322 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:11:33.532944 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:11:33.532910 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:11:46.433809 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:11:46.433768 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:11:56.234352 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:11:56.234319 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:12:03.669011 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:12:03.668978 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:12:12.232066 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:12:12.232025 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:12:20.229533 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:12:20.229493 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:12:36.627055 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:12:36.627022 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:12:45.129564 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:12:45.129526 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:12:50.610993 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:12:50.610968 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 21:12:50.613713 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:12:50.613687 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 21:12:53.722852 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:12:53.722817 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:13:02.536388 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:02.536355 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:13:25.628465 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:25.628429 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:13:38.831655 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:38.831624 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-78c99df468-bz4sz"] Apr 16 21:13:44.248403 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:44.248367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-764b68564c-75h4v_0907960e-97cf-48f2-a256-f59d92f13615/manager/0.log" Apr 16 21:13:44.715104 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:44.715078 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cd8df7dd5-t8gln_edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc/manager/0.log" Apr 16 21:13:45.577483 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.577453 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq_dace4755-9fcf-42d7-9ab4-e3a9d371f004/util/0.log" Apr 16 21:13:45.584253 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.584232 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq_dace4755-9fcf-42d7-9ab4-e3a9d371f004/pull/0.log" Apr 16 21:13:45.592224 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.592205 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq_dace4755-9fcf-42d7-9ab4-e3a9d371f004/extract/0.log" Apr 16 21:13:45.695711 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.695675 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk_53bbc7b6-a3e5-410e-9106-08828d972dee/util/0.log" Apr 16 21:13:45.701983 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.701949 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk_53bbc7b6-a3e5-410e-9106-08828d972dee/pull/0.log" Apr 16 21:13:45.707845 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.707823 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk_53bbc7b6-a3e5-410e-9106-08828d972dee/extract/0.log" Apr 16 21:13:45.815793 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.815761 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm_2dc44f77-16a9-4713-b59d-c07093e1b1f0/util/0.log" Apr 16 21:13:45.821907 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.821888 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm_2dc44f77-16a9-4713-b59d-c07093e1b1f0/pull/0.log" Apr 16 21:13:45.828454 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.828394 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm_2dc44f77-16a9-4713-b59d-c07093e1b1f0/extract/0.log" Apr 16 21:13:45.933491 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.933472 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm_b5808b89-67e6-4adf-886b-f3ff4499a1ff/util/0.log" Apr 16 21:13:45.939837 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.939817 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm_b5808b89-67e6-4adf-886b-f3ff4499a1ff/pull/0.log" Apr 16 21:13:45.948155 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:45.948137 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm_b5808b89-67e6-4adf-886b-f3ff4499a1ff/extract/0.log" Apr 16 21:13:46.301527 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:46.301501 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-648d5c98bc-q8pd6_03f33a24-6191-4f03-b0c4-200afd0eaee3/manager/0.log" Apr 16 21:13:46.411512 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:46.411493 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6cb54b5c86-whgnm_ed555409-3a9b-43c0-ba01-594cf026ffe0/kuadrant-console-plugin/0.log" Apr 16 21:13:46.519639 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:46.519597 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-d6c2b_3e772558-6975-4286-a0a0-4a7c45ecc923/registry-server/0.log" Apr 16 21:13:46.754632 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:46.754550 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-78c99df468-bz4sz_001cc2c6-4ef2-4879-8a91-ed2c6e58bc63/limitador/0.log" Apr 16 21:13:46.870325 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:46.870293 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-85c4996f8c-2h5w4_8fcfe231-6a16-491b-a4eb-f144cadc9fbe/manager/0.log" Apr 16 21:13:47.207765 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:47.207737 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59_059cd12f-f5b5-444d-9497-f0e292f1ed45/istio-proxy/0.log" Apr 16 21:13:47.668839 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:47.668809 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58b6f876-jvj29_b8fcb52d-f173-4e19-8a69-90a678a0164e/istio-proxy/0.log" Apr 16 21:13:47.802589 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:47.802563 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8c8485648-xl4zk_6721f3c5-b29f-4e7b-989f-9eff82d003b6/router/0.log" Apr 16 21:13:52.405954 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.405925 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gr7vb/must-gather-9kcmg"] Apr 16 21:13:52.406290 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.406265 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db048725-ff01-489c-8272-c4057dc83b88" containerName="cleanup" Apr 16 21:13:52.406290 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.406274 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db048725-ff01-489c-8272-c4057dc83b88" containerName="cleanup" Apr 16 21:13:52.406290 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.406283 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db048725-ff01-489c-8272-c4057dc83b88" containerName="cleanup" Apr 16 21:13:52.406290 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.406288 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db048725-ff01-489c-8272-c4057dc83b88" containerName="cleanup" Apr 16 21:13:52.406418 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.406298 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db048725-ff01-489c-8272-c4057dc83b88" containerName="cleanup" Apr 16 21:13:52.406418 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.406304 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db048725-ff01-489c-8272-c4057dc83b88" containerName="cleanup" Apr 16 21:13:52.406418 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.406353 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="db048725-ff01-489c-8272-c4057dc83b88" containerName="cleanup" Apr 16 21:13:52.406418 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.406360 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="db048725-ff01-489c-8272-c4057dc83b88" containerName="cleanup" Apr 16 21:13:52.406674 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.406469 2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="db048725-ff01-489c-8272-c4057dc83b88" containerName="cleanup" Apr 16 21:13:52.409356 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.409339 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gr7vb/must-gather-9kcmg" Apr 16 21:13:52.413026 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.413004 2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gr7vb\"/\"default-dockercfg-55qmz\"" Apr 16 21:13:52.414086 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.414068 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gr7vb\"/\"kube-root-ca.crt\"" Apr 16 21:13:52.414353 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.414333 2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gr7vb\"/\"openshift-service-ca.crt\"" Apr 16 21:13:52.428020 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.427999 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gr7vb/must-gather-9kcmg"] Apr 16 21:13:52.574739 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.574714 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9f2\" (UniqueName: \"kubernetes.io/projected/9e7762c8-61e1-48e5-b3d6-ee0450119a17-kube-api-access-rn9f2\") pod \"must-gather-9kcmg\" (UID: \"9e7762c8-61e1-48e5-b3d6-ee0450119a17\") " pod="openshift-must-gather-gr7vb/must-gather-9kcmg" Apr 16 21:13:52.574864 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.574757 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e7762c8-61e1-48e5-b3d6-ee0450119a17-must-gather-output\") pod \"must-gather-9kcmg\" (UID: \"9e7762c8-61e1-48e5-b3d6-ee0450119a17\") " pod="openshift-must-gather-gr7vb/must-gather-9kcmg" Apr 16 21:13:52.675735 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.675674 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9f2\" (UniqueName: \"kubernetes.io/projected/9e7762c8-61e1-48e5-b3d6-ee0450119a17-kube-api-access-rn9f2\") pod \"must-gather-9kcmg\" (UID: \"9e7762c8-61e1-48e5-b3d6-ee0450119a17\") " pod="openshift-must-gather-gr7vb/must-gather-9kcmg" Apr 16 21:13:52.675735 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.675715 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e7762c8-61e1-48e5-b3d6-ee0450119a17-must-gather-output\") pod \"must-gather-9kcmg\" (UID: \"9e7762c8-61e1-48e5-b3d6-ee0450119a17\") " pod="openshift-must-gather-gr7vb/must-gather-9kcmg" Apr 16 21:13:52.675998 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.675984 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e7762c8-61e1-48e5-b3d6-ee0450119a17-must-gather-output\") pod \"must-gather-9kcmg\" (UID: \"9e7762c8-61e1-48e5-b3d6-ee0450119a17\") " pod="openshift-must-gather-gr7vb/must-gather-9kcmg" Apr 16 21:13:52.684577 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.684558 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9f2\" (UniqueName: \"kubernetes.io/projected/9e7762c8-61e1-48e5-b3d6-ee0450119a17-kube-api-access-rn9f2\") pod \"must-gather-9kcmg\" (UID: \"9e7762c8-61e1-48e5-b3d6-ee0450119a17\") " pod="openshift-must-gather-gr7vb/must-gather-9kcmg" Apr 16 21:13:52.718237 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.718216 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gr7vb/must-gather-9kcmg" Apr 16 21:13:52.838666 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.838642 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gr7vb/must-gather-9kcmg"] Apr 16 21:13:52.840868 ip-10-0-142-90 kubenswrapper[2562]: W0416 21:13:52.840839 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e7762c8_61e1_48e5_b3d6_ee0450119a17.slice/crio-f4c6d2d2db52e689ea863263eb36cc7161f7ceaadb047895e2f8a043d798c3be WatchSource:0}: Error finding container f4c6d2d2db52e689ea863263eb36cc7161f7ceaadb047895e2f8a043d798c3be: Status 404 returned error can't find the container with id f4c6d2d2db52e689ea863263eb36cc7161f7ceaadb047895e2f8a043d798c3be Apr 16 21:13:52.842559 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:52.842539 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 21:13:53.007375 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:53.007285 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gr7vb/must-gather-9kcmg" event={"ID":"9e7762c8-61e1-48e5-b3d6-ee0450119a17","Type":"ContainerStarted","Data":"f4c6d2d2db52e689ea863263eb36cc7161f7ceaadb047895e2f8a043d798c3be"} Apr 16 21:13:54.017245 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:54.017164 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gr7vb/must-gather-9kcmg" event={"ID":"9e7762c8-61e1-48e5-b3d6-ee0450119a17","Type":"ContainerStarted","Data":"2be5dd378d9c48b5d2801334dcc58e0ab3631993a5f36bf3a58549cf30d8310b"} Apr 16 21:13:54.017245 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:54.017211 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gr7vb/must-gather-9kcmg" event={"ID":"9e7762c8-61e1-48e5-b3d6-ee0450119a17","Type":"ContainerStarted","Data":"947708e7ae36c8bb73d9f8acf426b5bab053a264c133ee4024f3e7cf125eb9bf"} Apr 16 21:13:54.034218 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:54.034169 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-gr7vb/must-gather-9kcmg" podStartSLOduration=1.154332147 podStartE2EDuration="2.034154534s" podCreationTimestamp="2026-04-16 21:13:52 +0000 UTC" firstStartedPulling="2026-04-16 21:13:52.842677811 +0000 UTC m=+2163.018042056" lastFinishedPulling="2026-04-16 21:13:53.722500197 +0000 UTC m=+2163.897864443" observedRunningTime="2026-04-16 21:13:54.033380984 +0000 UTC m=+2164.208745249" watchObservedRunningTime="2026-04-16 21:13:54.034154534 +0000 UTC m=+2164.209518825" Apr 16 21:13:55.319042 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:55.319012 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mmchs_3fc39fb1-0592-4764-a26a-7d30d295d17f/global-pull-secret-syncer/0.log" Apr 16 21:13:55.373966 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:55.373927 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4jtmh_a66139a0-edaa-46ff-b00c-9ed2765a0ae6/konnectivity-agent/0.log" Apr 16 21:13:55.509521 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:55.509489 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-90.ec2.internal_ffbe1178dcecf35d909f25cd02be28cb/haproxy/0.log" Apr 16 21:13:59.120224 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.120136 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq_dace4755-9fcf-42d7-9ab4-e3a9d371f004/extract/0.log" Apr 16 21:13:59.146151 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.146118 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq_dace4755-9fcf-42d7-9ab4-e3a9d371f004/util/0.log" Apr 16 21:13:59.168844 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.168767 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759tq2kq_dace4755-9fcf-42d7-9ab4-e3a9d371f004/pull/0.log" Apr 16 21:13:59.218560 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.218493 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk_53bbc7b6-a3e5-410e-9106-08828d972dee/extract/0.log" Apr 16 21:13:59.254977 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.254952 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk_53bbc7b6-a3e5-410e-9106-08828d972dee/util/0.log" Apr 16 21:13:59.310774 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.310727 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0glzdk_53bbc7b6-a3e5-410e-9106-08828d972dee/pull/0.log" Apr 16 21:13:59.345543 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.345517 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm_2dc44f77-16a9-4713-b59d-c07093e1b1f0/extract/0.log" Apr 16 21:13:59.374436 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.374389 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm_2dc44f77-16a9-4713-b59d-c07093e1b1f0/util/0.log" Apr 16 21:13:59.400006 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.399964 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed737rxrm_2dc44f77-16a9-4713-b59d-c07093e1b1f0/pull/0.log" Apr 16 21:13:59.428074 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.427981 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm_b5808b89-67e6-4adf-886b-f3ff4499a1ff/extract/0.log" Apr 16 21:13:59.452175 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.452143 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm_b5808b89-67e6-4adf-886b-f3ff4499a1ff/util/0.log" Apr 16 21:13:59.477057 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.477031 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15gnqm_b5808b89-67e6-4adf-886b-f3ff4499a1ff/pull/0.log" Apr 16 21:13:59.722267 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.722188 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-648d5c98bc-q8pd6_03f33a24-6191-4f03-b0c4-200afd0eaee3/manager/0.log" Apr 16 21:13:59.769531 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.769496 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6cb54b5c86-whgnm_ed555409-3a9b-43c0-ba01-594cf026ffe0/kuadrant-console-plugin/0.log" Apr 16 21:13:59.825657 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.825624 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-d6c2b_3e772558-6975-4286-a0a0-4a7c45ecc923/registry-server/0.log" Apr 16 21:13:59.932022 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:13:59.931996 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-78c99df468-bz4sz_001cc2c6-4ef2-4879-8a91-ed2c6e58bc63/limitador/0.log" Apr 16 21:14:00.010356 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:00.010277 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-85c4996f8c-2h5w4_8fcfe231-6a16-491b-a4eb-f144cadc9fbe/manager/0.log" Apr 16 21:14:01.801431 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:01.801402 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f6fb6ccd-9tk8n_2e40448a-e313-48bc-9a04-1475811e4924/metrics-server/0.log" Apr 16 21:14:01.935883 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:01.935811 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cgclg_8b5670c6-cfdc-4985-925a-8ab32521a7ce/node-exporter/0.log" Apr 16 21:14:01.957657 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:01.957630 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cgclg_8b5670c6-cfdc-4985-925a-8ab32521a7ce/kube-rbac-proxy/0.log" Apr 16 21:14:01.978646 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:01.978622 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cgclg_8b5670c6-cfdc-4985-925a-8ab32521a7ce/init-textfile/0.log" Apr 16 21:14:02.094279 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:02.094193 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9d44df66c-m48sw_47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116/kube-rbac-proxy-main/0.log" Apr 16 21:14:02.120060 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:02.120008 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9d44df66c-m48sw_47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116/kube-rbac-proxy-self/0.log" Apr 16 21:14:02.149674 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:02.149621 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9d44df66c-m48sw_47f2c6c8-a0c6-4fb4-8cb8-ac5b5d7e6116/openshift-state-metrics/0.log" Apr 16 21:14:04.127626 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.127570 2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh"] Apr 16 21:14:04.134584 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.134556 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.141617 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.141581 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh"] Apr 16 21:14:04.181215 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.181185 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/2.log" Apr 16 21:14:04.187321 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.187289 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-9d4b6777b-s4bp5_dc0a46ee-2e45-4265-8a43-5491b384d9f7/console-operator/3.log" Apr 16 21:14:04.288565 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.288528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-proc\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.288854 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.288831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-sys\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.289052 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.289037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-podres\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.289183 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.289167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8qjw\" (UniqueName: \"kubernetes.io/projected/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-kube-api-access-t8qjw\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.289348 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.289331 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-lib-modules\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.390762 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.390669 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-proc\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.390762 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.390725 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-sys\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.391024 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.390776 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-podres\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.391024 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.390805 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-proc\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.391024 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.390813 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t8qjw\" (UniqueName: \"kubernetes.io/projected/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-kube-api-access-t8qjw\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.391024 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.390885 2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-lib-modules\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.391243 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.391090 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-lib-modules\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.391243 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.391155 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-sys\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.391243 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.391170 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-podres\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.401177 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.401142 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8qjw\" (UniqueName: \"kubernetes.io/projected/d02d0909-9e9b-4eb8-9987-6fc37b9741ef-kube-api-access-t8qjw\") pod \"perf-node-gather-daemonset-rrzqh\" (UID: \"d02d0909-9e9b-4eb8-9987-6fc37b9741ef\") " pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.448093 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.448063 2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:04.607080 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.606995 2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh"] Apr 16 21:14:04.697480 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:04.697456 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9c4b4cfdd-njfxv_12424dd7-aefc-476f-b59f-65b8a289c774/console/0.log" Apr 16 21:14:05.070438 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:05.070397 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" event={"ID":"d02d0909-9e9b-4eb8-9987-6fc37b9741ef","Type":"ContainerStarted","Data":"4507c6ac262bff4dbe79430bccc59fc1386040e131856927c7216e63ebe63083"} Apr 16 21:14:05.070438 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:05.070443 2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" event={"ID":"d02d0909-9e9b-4eb8-9987-6fc37b9741ef","Type":"ContainerStarted","Data":"53582b9f6489125091a846ed22f6c702fa45545a32517bee51e4399fdf1195ff"} Apr 16 21:14:05.070696 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:05.070462 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:05.101005 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:05.100949 2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" podStartSLOduration=1.100929695 podStartE2EDuration="1.100929695s" podCreationTimestamp="2026-04-16 21:14:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 21:14:05.100514419 +0000 UTC m=+2175.275878686" watchObservedRunningTime="2026-04-16 21:14:05.100929695 +0000 UTC m=+2175.276293962" Apr 16 21:14:06.133309 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:06.133285 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vxpfw_c72f51ec-57bd-429f-85d6-aa06c2c66728/dns/0.log" Apr 16 21:14:06.155319 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:06.155300 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vxpfw_c72f51ec-57bd-429f-85d6-aa06c2c66728/kube-rbac-proxy/0.log" Apr 16 21:14:06.188270 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:06.188246 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jxb6_24dbfe36-1150-4ee2-b906-d2d5107ac31c/dns-node-resolver/0.log" Apr 16 21:14:06.772327 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:06.772300 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qhhws_427d721c-2662-4922-a063-4b44f92e8ca2/node-ca/0.log" Apr 16 21:14:07.523363 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:07.523329 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5cb8b776cfc7r59_059cd12f-f5b5-444d-9497-f0e292f1ed45/istio-proxy/0.log" Apr 16 21:14:07.824730 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:07.824701 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58b6f876-jvj29_b8fcb52d-f173-4e19-8a69-90a678a0164e/istio-proxy/0.log" Apr 16 21:14:07.865313 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:07.865281 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8c8485648-xl4zk_6721f3c5-b29f-4e7b-989f-9eff82d003b6/router/0.log" Apr 16 21:14:08.417762 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:08.417734 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qtfdj_6781237f-45c1-4874-a81d-43042624470f/serve-healthcheck-canary/0.log" Apr 16 21:14:09.023378 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:09.023344 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-scchg_873664c9-3817-4764-8f81-37ec78380ce2/kube-rbac-proxy/0.log" Apr 16 21:14:09.047007 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:09.046977 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-scchg_873664c9-3817-4764-8f81-37ec78380ce2/exporter/0.log" Apr 16 21:14:09.067251 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:09.067227 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-scchg_873664c9-3817-4764-8f81-37ec78380ce2/extractor/0.log" Apr 16 21:14:11.084539 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:11.084513 2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gr7vb/perf-node-gather-daemonset-rrzqh" Apr 16 21:14:11.148030 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:11.147999 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-764b68564c-75h4v_0907960e-97cf-48f2-a256-f59d92f13615/manager/0.log" Apr 16 21:14:11.282232 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:11.282206 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cd8df7dd5-t8gln_edf62ef4-faa3-4aec-82a5-54ddc0e8bcfc/manager/0.log" Apr 16 21:14:12.446634 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:12.446585 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5494fc4578-7c4xp_6dad5e38-aa96-4695-9fb2-6f876421925f/manager/0.log" Apr 16 21:14:17.413825 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:17.413792 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6769c5d45-8xsb6_b064c14c-045d-4f5f-b252-0ddbff4e03db/kube-storage-version-migrator-operator/1.log" Apr 16 21:14:17.414647 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:17.414625 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6769c5d45-8xsb6_b064c14c-045d-4f5f-b252-0ddbff4e03db/kube-storage-version-migrator-operator/0.log" Apr 16 21:14:19.219826 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:19.219795 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2b6m_149ff0b8-9dee-4e72-b694-9ac834034877/kube-multus-additional-cni-plugins/0.log" Apr 16 21:14:19.249733 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:19.249709 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2b6m_149ff0b8-9dee-4e72-b694-9ac834034877/egress-router-binary-copy/0.log" Apr 16 21:14:19.290326 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:19.290299 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2b6m_149ff0b8-9dee-4e72-b694-9ac834034877/cni-plugins/0.log" Apr 16 21:14:19.320424 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:19.320403 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2b6m_149ff0b8-9dee-4e72-b694-9ac834034877/bond-cni-plugin/0.log" Apr 16 21:14:19.343430 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:19.343409 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2b6m_149ff0b8-9dee-4e72-b694-9ac834034877/routeoverride-cni/0.log" Apr 16 21:14:19.381399 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:19.381375 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2b6m_149ff0b8-9dee-4e72-b694-9ac834034877/whereabouts-cni-bincopy/0.log" Apr 16 21:14:19.412514 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:19.412488 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2b6m_149ff0b8-9dee-4e72-b694-9ac834034877/whereabouts-cni/0.log" Apr 16 21:14:19.449885 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:19.449868 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bcnw8_005c632a-67fa-4e6a-9d1b-ab17c5f0e48d/kube-multus/0.log" Apr 16 21:14:19.627268 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:19.627245 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w8xr8_b8187ddd-1510-4903-bc28-7206752133c5/network-metrics-daemon/0.log" Apr 16 21:14:19.658969 ip-10-0-142-90 kubenswrapper[2562]: I0416 21:14:19.658948 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w8xr8_b8187ddd-1510-4903-bc28-7206752133c5/kube-rbac-proxy/0.log"