Jul 22 11:34:25.418092 ip-10-0-132-210 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 11:34:25.418107 ip-10-0-132-210 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 11:34:25.418118 ip-10-0-132-210 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 11:34:25.418438 ip-10-0-132-210 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 11:34:34.800195 ip-10-0-132-210 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 11:34:34.800211 ip-10-0-132-210 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot acd85a431bda4ac896d3b845ef05926c -- Jul 22 11:36:02.545720 ip-10-0-132-210 systemd[1]: Starting Kubernetes Kubelet... Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.966611 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971297 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971310 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971313 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971316 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971319 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971321 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971324 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:36:03.319700 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971328 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:36:03.044075 ip-10-0-132-210 systemd[1]: Started Kubernetes Kubelet. Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971332 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971335 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971337 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971340 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971342 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971345 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971348 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971351 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971353 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971356 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971359 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971361 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971365 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971367 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971379 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971394 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971397 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971405 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971408 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:36:03.332947 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971411 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971413 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971416 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971419 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971421 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971424 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971427 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971430 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971432 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971435 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971438 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971442 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971446 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971448 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971451 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971454 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971456 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971459 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971461 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:36:03.335467 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971464 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971467 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971470 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971472 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971475 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971477 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971480 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971482 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971485 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971487 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971490 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971492 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971495 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971497 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971500 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971503 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971506 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971508 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971511 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971514 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:36:03.337165 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971516 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971518 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971521 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971523 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971526 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971528 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971531 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971533 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971536 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971538 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971541 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971543 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971545 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971554 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971557 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971559 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971561 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971565 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971568 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:36:03.339913 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971572 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971576 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971579 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971581 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971584 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971587 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971589 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971593 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971596 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971599 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971601 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971604 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971607 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.971609 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972381 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972404 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972407 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972411 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972414 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972416 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:36:03.342623 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972419 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972421 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972424 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972426 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972429 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972431 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972434 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972442 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972445 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972447 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972450 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972452 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972455 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972457 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972460 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972462 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972464 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972467 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972469 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:36:03.345218 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972472 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972475 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972479 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972481 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972484 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972486 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972489 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972491 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972493 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972496 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972499 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972503 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972506 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972508 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972511 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972513 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972516 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972518 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972521 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972523 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:36:03.346132 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972525 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972529 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972531 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972534 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972536 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972539 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972541 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972543 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972546 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972549 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972551 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972555 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972557 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972559 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972565 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972568 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972570 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972573 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972576 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972578 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:36:03.348126 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972581 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972583 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972586 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972589 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972592 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972594 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972597 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972599 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972602 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972604 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972607 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972609 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972612 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972614 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972617 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972620 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972623 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972626 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972628 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972630 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:36:03.349791 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972633 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972637 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972641 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972643 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972646 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972649 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972654 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972657 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972660 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972663 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972666 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972669 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972671 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.972674 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973036 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973040 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973043 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973046 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973049 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:36:03.350786 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973051 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973054 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973061 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973063 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973066 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973068 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973071 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973074 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973077 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973080 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973082 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973084 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973087 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973089 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973092 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973094 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973097 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973099 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973102 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973106 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:36:03.351750 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973109 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973111 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973113 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973116 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973119 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973122 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973124 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973127 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973129 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973132 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973134 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973137 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973139 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973142 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973144 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973147 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973149 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973153 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973157 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:36:03.352452 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973160 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973165 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973167 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973170 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973172 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973175 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973177 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973180 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973182 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973184 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973187 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973189 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973194 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973197 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973200 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973203 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973205 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973208 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973210 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973213 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:36:03.353138 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973215 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973218 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973220 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973223 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973225 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973228 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973232 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973235 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973239 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973242 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973244 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973247 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973250 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973253 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973256 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973258 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973261 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973264 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973266 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973268 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:36:03.353816 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973271 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973273 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973275 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973278 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973281 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973283 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973286 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973288 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973291 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973293 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973296 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973298 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973300 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973303 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973306 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973375 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973399 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973406 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973411 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973415 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973418 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 11:36:03.354707 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973424 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973429 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973432 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973434 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973440 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973443 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973446 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973450 2579 flags.go:64] FLAG: --cgroup-root="" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973453 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973456 2579 flags.go:64] FLAG: --client-ca-file="" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973458 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973461 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973465 2579 flags.go:64] FLAG: --cluster-domain="" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973468 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973471 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973476 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973479 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973482 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973486 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973488 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973492 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973495 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973498 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973501 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 11:36:03.355318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973504 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973507 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973511 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973513 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973516 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973519 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973522 2579 flags.go:64] FLAG: --enable-server="true" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973525 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973529 2579 flags.go:64] FLAG: --event-burst="100" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973532 2579 flags.go:64] FLAG: --event-qps="50" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973535 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973537 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973545 2579 flags.go:64] FLAG: --eviction-hard="" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973548 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973551 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973554 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973557 2579 flags.go:64] FLAG: --eviction-soft="" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973560 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973563 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973566 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973569 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973572 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973574 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973578 2579 flags.go:64] FLAG: --feature-gates="" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973582 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 11:36:03.356033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973585 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973588 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973591 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973594 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973597 2579 flags.go:64] FLAG: --help="false" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973600 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973603 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973606 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973609 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973612 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973615 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973618 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973620 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973623 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973626 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973629 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973632 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973634 2579 flags.go:64] FLAG: --kube-reserved="" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973637 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973641 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973644 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973647 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973650 2579 flags.go:64] FLAG: --lock-file="" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973653 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 11:36:03.356778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973656 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973658 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973663 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973666 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973669 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973671 2579 flags.go:64] FLAG: --logging-format="text" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973676 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973679 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973682 2579 flags.go:64] FLAG: --manifest-url="" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973685 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973689 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973692 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973695 2579 flags.go:64] FLAG: --max-pods="110" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973698 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973701 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973704 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973707 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973710 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973712 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973715 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973722 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973725 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973728 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 11:36:03.357460 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973731 2579 flags.go:64] FLAG: --pod-cidr="" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973734 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973739 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973741 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973746 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973749 2579 flags.go:64] FLAG: --port="10250" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973751 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973754 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b50444d6c3446521" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973757 2579 flags.go:64] FLAG: --qos-reserved="" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973760 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973763 2579 flags.go:64] FLAG: --register-node="true" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973766 2579 flags.go:64] FLAG: --register-with-taints="" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973770 2579 flags.go:64] FLAG: --registry-burst="10" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973772 2579 flags.go:64] FLAG: --registry-qps="5" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973775 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973777 2579 flags.go:64] FLAG: --reserved-memory="" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973782 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973785 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973788 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973790 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973793 2579 flags.go:64] FLAG: --runonce="false" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973796 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973798 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973801 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973804 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 11:36:03.360397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973806 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973809 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973812 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973816 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973818 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973821 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973824 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973827 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973829 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973832 2579 flags.go:64] FLAG: --system-cgroups="" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973835 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973842 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973844 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973847 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973850 2579 flags.go:64] FLAG: --tls-min-version="" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973853 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973855 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973859 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973862 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973865 2579 flags.go:64] FLAG: --v="2" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973869 2579 flags.go:64] FLAG: --version="false" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973872 2579 flags.go:64] FLAG: --vmodule="" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973875 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.973880 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973988 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:36:03.361277 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973991 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973995 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.973998 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974001 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974004 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974007 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974009 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974012 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974014 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974017 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974019 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974022 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974024 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974027 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974029 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974031 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974034 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974036 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974041 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:36:03.368001 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974043 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974046 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974048 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974051 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974053 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974055 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974058 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974061 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974063 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974065 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974068 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974072 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974074 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974077 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974081 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974085 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974088 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974090 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974093 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:36:03.368696 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974095 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974098 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974101 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974103 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974106 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974108 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974111 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974113 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974116 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974118 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974121 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974123 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974128 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974131 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974133 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974136 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974138 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974141 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974143 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:36:03.369416 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974147 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974151 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974153 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974156 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974159 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974163 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974166 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974169 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974172 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974175 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974177 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974179 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974182 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974185 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974187 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974190 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974192 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974195 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974197 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974200 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:36:03.370089 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974202 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974205 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974207 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974210 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974212 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974216 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974218 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974221 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974224 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974226 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974229 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974232 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974234 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974237 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974241 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974245 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974247 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974251 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974254 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974256 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:36:03.370793 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.974259 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.974264 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.981174 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.981189 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.981205 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.981215 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981280 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981286 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981292 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981297 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981300 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981303 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981306 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981308 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981311 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:36:03.426073 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981314 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981316 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981319 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981321 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981324 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981327 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981329 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981332 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981335 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981337 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981340 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981344 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981348 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981351 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981355 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981359 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981361 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981364 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981366 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:36:03.426705 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981369 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981372 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981374 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981377 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981380 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981400 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981406 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981408 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981411 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981413 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981416 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981419 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981422 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981424 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981427 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981429 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981432 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981438 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981440 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981443 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:36:03.427363 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981445 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981448 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981450 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981453 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981455 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981458 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981460 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981463 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981465 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981468 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981470 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981473 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981475 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981478 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981481 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981484 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981486 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981488 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981492 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:36:03.428024 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981494 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981497 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981499 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981502 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981504 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981507 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981509 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981511 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981514 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981516 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981520 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981523 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981525 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981527 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981530 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981532 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981535 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981537 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981540 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981542 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:36:03.428682 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981545 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981547 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981551 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981554 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981557 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981560 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981563 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981566 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981575 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981578 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981581 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981585 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.981590 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981735 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981740 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 11:36:03.429328 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981742 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981745 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981748 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981750 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981753 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981755 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981758 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981760 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981764 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981766 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981769 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981771 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981774 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981776 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981779 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981781 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981784 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981786 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981788 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981791 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 11:36:03.429999 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981793 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981796 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981798 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981801 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981803 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981805 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981815 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981819 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981823 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981826 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981828 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981831 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981833 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981836 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981838 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981840 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981843 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981845 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981848 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 11:36:03.430669 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981850 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981854 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981856 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981859 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981861 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981864 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981866 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981868 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981871 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981873 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981875 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981878 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981880 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981883 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981885 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981888 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981890 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981892 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981895 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981897 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 11:36:03.431329 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981905 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981909 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981912 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981914 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981916 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981919 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981921 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981924 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981928 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981931 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981933 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981936 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981939 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981943 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981947 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981950 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981952 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981955 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981958 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 11:36:03.431990 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981961 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981963 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981966 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981969 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981971 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981974 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981976 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981979 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981982 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981984 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981987 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981989 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981992 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.981994 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.982003 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.982006 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.982008 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.982011 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 11:36:03.432653 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:02.982013 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.982018 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.982156 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.984261 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.985511 2579 server.go:1019] "Starting client certificate rotation" Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.985612 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.985661 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.990635 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:02.995025 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.005252 2579 log.go:25] "Validated CRI v1 runtime API" Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.010199 2579 log.go:25] "Validated CRI v1 image API" Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.011281 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 11:36:03.433325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.014627 2579 fs.go:135] Filesystem UUIDs: map[768f6f98-0f6e-4b91-b8d4-b02e56bdcc8d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d50a85f3-aa2f-461d-bc86-4b09c2f4cf4f:/dev/nvme0n1p4] Jul 22 11:36:03.433765 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.014643 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 11:36:03.433765 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.021039 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.021177 2579 manager.go:216] Machine: {Timestamp:2026-07-22 11:36:03.019614198 +0000 UTC m=+0.349397699 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199270 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c3b517a35165a5d21cf37d077d69 SystemUUID:ec22c3b5-17a3-5165-a5d2-1cf37d077d69 BootID:acd85a43-1bda-4ac8-96d3-b845ef05926c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:33:17:ea:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:33:17:ea:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:ff:3f:dc:80:35 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.021283 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.021417 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.025149 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.025177 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.026395 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.026404 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.026412 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.027113 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.028363 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.031248 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.031361 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.031410 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.031420 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.031438 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.031447 2579 kubelet.go:398] "Adding apiserver pod source" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.031469 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 11:36:03.434216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.031905 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.031922 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.035249 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.037137 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.037164 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038506 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038532 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038543 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038553 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038567 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038595 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038613 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038624 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038635 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038648 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038674 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.038682 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.039606 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.039616 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.040605 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.040611 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.043097 2579 server.go:1268] "Started kubelet" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.043173 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.043313 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.043698 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.048134 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.048156 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.050621 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.051116 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.055103 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.055109 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.055180 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.055758 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.055761 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.055786 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.055886 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.055894 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.055939 2579 factory.go:55] Registering systemd factory Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.055955 2579 factory.go:223] Registration of the systemd container factory successfully Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.056207 2579 factory.go:153] Registering CRI-O factory Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.056220 2579 factory.go:223] Registration of the crio container factory successfully Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.056269 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.056298 2579 factory.go:103] Registering Raw factory Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.056324 2579 manager.go:1196] Started watching for new ooms in manager Jul 22 11:36:03.437358 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.056698 2579 manager.go:349] Starting recovery of all containers Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.055943 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-210.ec2.internal.18c499884da8cbb9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-210.ec2.internal,UID:ip-10-0-132-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-210.ec2.internal,},FirstTimestamp:2026-07-22 11:36:03.043068857 +0000 UTC m=+0.372852340,LastTimestamp:2026-07-22 11:36:03.043068857 +0000 UTC m=+0.372852340,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-210.ec2.internal,}" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.062527 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.062645 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.067046 2579 manager.go:354] Recovery completed Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.070292 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nrds5" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.073450 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.075404 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.075428 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.075445 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.075953 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.075960 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.075965 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.075974 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.076345 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nrds5" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.076622 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.076651 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.076660 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.077319 2579 policy_none.go:49] "None policy: Start" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.077341 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.077355 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.077576 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-210.ec2.internal.18c499884f965c35 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-210.ec2.internal,UID:ip-10-0-132-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-210.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-210.ec2.internal,},FirstTimestamp:2026-07-22 11:36:03.075415093 +0000 UTC m=+0.405198574,LastTimestamp:2026-07-22 11:36:03.075415093 +0000 UTC m=+0.405198574,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-210.ec2.internal,}" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.078352 2579 policy_none.go:47] "Start" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.155363 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.158942 2579 manager.go:337] "Starting Device Plugin manager" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.158972 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.158984 2579 server.go:87] "Starting device plugin registration server" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.159358 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.159375 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.159479 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.159556 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.159564 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 11:36:03.438703 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.160107 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.160159 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.167966 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.169152 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.169190 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.169213 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.169253 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.171473 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.259934 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.261179 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.261207 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.261220 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.261248 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.269575 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal"] Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.269631 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.269669 2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-210.ec2.internal\" not found" node="ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.270424 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.270451 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.270468 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.270856 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.270874 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-210.ec2.internal\": node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.271746 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.271916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.271941 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.272456 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.272460 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.272480 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.272484 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.272489 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.272498 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.273585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.273606 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.274253 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientMemory" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.274272 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.274285 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeHasSufficientPID" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.294969 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-210.ec2.internal\" not found" node="ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.298475 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:03.500360 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.298919 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-210.ec2.internal\" not found" node="ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.605891 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.545016 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:03.605891 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.545100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab9df11ccd03f66e0259d1240fa3368b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal\" (UID: \"ab9df11ccd03f66e0259d1240fa3368b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.605891 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.545127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6115ca8597103d232ce4b4815a5dee79-config\") pod \"kube-apiserver-proxy-ip-10-0-132-210.ec2.internal\" (UID: \"6115ca8597103d232ce4b4815a5dee79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.605891 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.545144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab9df11ccd03f66e0259d1240fa3368b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal\" (UID: \"ab9df11ccd03f66e0259d1240fa3368b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.746955 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.645685 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:03.746955 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.645704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab9df11ccd03f66e0259d1240fa3368b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal\" (UID: \"ab9df11ccd03f66e0259d1240fa3368b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.746955 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.645735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6115ca8597103d232ce4b4815a5dee79-config\") pod \"kube-apiserver-proxy-ip-10-0-132-210.ec2.internal\" (UID: \"6115ca8597103d232ce4b4815a5dee79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.746955 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.645749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab9df11ccd03f66e0259d1240fa3368b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal\" (UID: \"ab9df11ccd03f66e0259d1240fa3368b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.746955 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.645790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab9df11ccd03f66e0259d1240fa3368b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal\" (UID: \"ab9df11ccd03f66e0259d1240fa3368b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.746955 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.645805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6115ca8597103d232ce4b4815a5dee79-config\") pod \"kube-apiserver-proxy-ip-10-0-132-210.ec2.internal\" (UID: \"6115ca8597103d232ce4b4815a5dee79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.746955 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.645804 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab9df11ccd03f66e0259d1240fa3368b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal\" (UID: \"ab9df11ccd03f66e0259d1240fa3368b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.746955 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.746061 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:03.868782 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.846513 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:03.986143 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.898873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.986143 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.901641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal" Jul 22 11:36:03.986143 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:03.947077 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:03.986143 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.985261 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 11:36:03.986143 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:03.985463 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 11:36:05.182100 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:05.078596 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:05.182100 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:05.115188 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 11:31:04 +0000 UTC" deadline="2028-03-19 19:02:45.078938928 +0000 UTC" Jul 22 11:36:05.182100 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:05.115270 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14551h26m39.963678496s" Jul 22 11:36:05.182100 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:05.178142 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:36:05.182100 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:05.179388 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:05.323395 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:05.282764 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:05.452717 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:05.383185 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:05.580322 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:05.483632 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:05.580322 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:05.548387 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9df11ccd03f66e0259d1240fa3368b.slice/crio-d209f1e882234b0772fb27d76fb99c4ba6804f1a97a334779610de7370de0d6f WatchSource:0}: Error finding container d209f1e882234b0772fb27d76fb99c4ba6804f1a97a334779610de7370de0d6f: Status 404 returned error can't find the container with id d209f1e882234b0772fb27d76fb99c4ba6804f1a97a334779610de7370de0d6f Jul 22 11:36:05.580322 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:05.558494 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6115ca8597103d232ce4b4815a5dee79.slice/crio-026801c1329b1eebca180e87c6e5a321eb80433ef6ddcd5db0b6510be2b7ca2c WatchSource:0}: Error finding container 026801c1329b1eebca180e87c6e5a321eb80433ef6ddcd5db0b6510be2b7ca2c: Status 404 returned error can't find the container with id 026801c1329b1eebca180e87c6e5a321eb80433ef6ddcd5db0b6510be2b7ca2c Jul 22 11:36:05.580322 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:05.559258 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:36:05.584268 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:05.584164 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-210.ec2.internal\" not found" Jul 22 11:36:05.726902 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:05.594009 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:36:05.726902 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:05.596977 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:36:05.726902 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:05.686639 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" Jul 22 11:36:05.726902 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:05.701105 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 11:36:05.726902 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:05.702267 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal" Jul 22 11:36:05.726902 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:05.713189 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 11:36:06.064536 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.063566 2579 apiserver.go:52] "Watching apiserver" Jul 22 11:36:06.074384 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.073010 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 11:36:06.074384 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.073409 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-gf2wn","openshift-ovn-kubernetes/ovnkube-node-5sctd","kube-system/konnectivity-agent-4vndv","kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5","openshift-cluster-node-tuning-operator/tuned-42rnz","openshift-dns/node-resolver-ndr58","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal","openshift-multus/multus-additional-cni-plugins-dbflp","kube-system/global-pull-secret-syncer-gcrmm","openshift-image-registry/node-ca-mwg4w","openshift-multus/multus-lmkbc","openshift-multus/network-metrics-daemon-x92mg","openshift-network-diagnostics/network-check-target-j7pfq"] Jul 22 11:36:06.077429 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.075188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:06.077429 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.075269 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:06.077429 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.077321 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:06.077633 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.077425 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.078599 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.081869 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.081957 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vcgkl\"" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.082287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.082375 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.083692 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.084002 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.085657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.086408 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.086578 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.086616 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ns84q\"" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.086756 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.086901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087080 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0-konnectivity-ca\") pod \"konnectivity-agent-4vndv\" (UID: \"1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0\") " pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z85n\" (UniqueName: \"kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n\") pod \"network-check-target-j7pfq\" (UID: \"60af647a-0dc7-4414-8cac-a61393b4cb51\") " pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-device-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-sys-fs\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d93eebd-7f1a-48fd-a0e3-51fe4ab46250-tmp-dir\") pod \"node-resolver-ndr58\" (UID: \"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250\") " pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0-agent-certs\") pod \"konnectivity-agent-4vndv\" (UID: \"1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0\") " pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-registration-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.147810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087682 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-socket-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95n9j\" (UniqueName: \"kubernetes.io/projected/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-kube-api-access-95n9j\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.087742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d93eebd-7f1a-48fd-a0e3-51fe4ab46250-hosts-file\") pod \"node-resolver-ndr58\" (UID: \"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250\") " pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.088797 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.089548 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.089554 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x5svw\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.089635 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.089956 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.089974 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.090144 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.090341 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.090693 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-btsgc\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.091065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.092345 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.092534 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.093328 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.093418 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dxk8z\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.093788 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.094047 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.094205 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.094452 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.094507 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-29nnd\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.094640 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.094768 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.095079 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.095085 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-chbgp\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.095255 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.095337 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.095371 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.095400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.095914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.099074 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.099082 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zjk8c\"" Jul 22 11:36:06.149054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.099363 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 11:36:06.151805 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.099571 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 11:36:06.151805 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.099594 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-796d8\"" Jul 22 11:36:06.151805 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.099601 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 11:36:06.151805 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.101864 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 11:36:06.151805 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.122979 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5nspb" Jul 22 11:36:06.151805 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.130885 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5nspb" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.187796 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.187955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1aca7ae8-e7fc-417d-840b-a7fa1155723f-kubelet-config\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.187986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-var-lib-cni-multus\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-ovn-node-metrics-cert\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-cnibin\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntn6k\" (UniqueName: \"kubernetes.io/projected/e9412ed9-53f3-4469-99aa-119dab079331-kube-api-access-ntn6k\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-systemd-units\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-run-netns\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-ovnkube-config\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-ovnkube-script-lib\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11037cad-c16a-4b04-b5ee-1287f5caef55-serviceca\") pod \"node-ca-mwg4w\" (UID: \"11037cad-c16a-4b04-b5ee-1287f5caef55\") " pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z85n\" (UniqueName: \"kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n\") pod \"network-check-target-j7pfq\" (UID: \"60af647a-0dc7-4414-8cac-a61393b4cb51\") " pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1aca7ae8-e7fc-417d-840b-a7fa1155723f-dbus\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11037cad-c16a-4b04-b5ee-1287f5caef55-host\") pod \"node-ca-mwg4w\" (UID: \"11037cad-c16a-4b04-b5ee-1287f5caef55\") " pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-sys-fs\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.221510 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-os-release\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghzh\" (UniqueName: \"kubernetes.io/projected/787909e2-590a-44e5-baa0-d5d7b711aa6b-kube-api-access-tghzh\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0-agent-certs\") pod \"konnectivity-agent-4vndv\" (UID: \"1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0\") " pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188386 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-var-lib-kubelet\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wxs\" (UniqueName: \"kubernetes.io/projected/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-kube-api-access-p8wxs\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188461 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-system-cni-dir\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-sys-fs\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9412ed9-53f3-4469-99aa-119dab079331-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-conf-dir\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-run-multus-certs\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-modprobe-d\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlq8h\" (UniqueName: \"kubernetes.io/projected/64781c58-80c2-4b43-bc93-8bd1a7df80ca-kube-api-access-hlq8h\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-run-netns\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-etc-openvswitch\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.222633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-run-openvswitch\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.188996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b0b3fa4-bc82-4499-a622-3fb464453a1c-iptables-alerter-script\") pod \"iptables-alerter-gf2wn\" (UID: \"3b0b3fa4-bc82-4499-a622-3fb464453a1c\") " pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0-konnectivity-ca\") pod \"konnectivity-agent-4vndv\" (UID: \"1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0\") " pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-run\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189108 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-system-cni-dir\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-log-socket\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d93eebd-7f1a-48fd-a0e3-51fe4ab46250-tmp-dir\") pod \"node-resolver-ndr58\" (UID: \"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250\") " pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-sysconfig\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189540 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-host\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-cni-dir\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-var-lib-kubelet\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0-konnectivity-ca\") pod \"konnectivity-agent-4vndv\" (UID: \"1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0\") " pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-cni-netd\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-registration-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpt4g\" (UniqueName: \"kubernetes.io/projected/11037cad-c16a-4b04-b5ee-1287f5caef55-kube-api-access-dpt4g\") pod \"node-ca-mwg4w\" (UID: \"11037cad-c16a-4b04-b5ee-1287f5caef55\") " pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.228304 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-socket-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95n9j\" (UniqueName: \"kubernetes.io/projected/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-kube-api-access-95n9j\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-cnibin\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-socket-dir-parent\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-etc-kubernetes\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-sysctl-d\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-systemd\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-os-release\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d93eebd-7f1a-48fd-a0e3-51fe4ab46250-tmp-dir\") pod \"node-resolver-ndr58\" (UID: \"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250\") " pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.189931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/787909e2-590a-44e5-baa0-d5d7b711aa6b-cni-binary-copy\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-registration-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-cni-bin\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsn98\" (UniqueName: \"kubernetes.io/projected/3d93eebd-7f1a-48fd-a0e3-51fe4ab46250-kube-api-access-jsn98\") pod \"node-resolver-ndr58\" (UID: \"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250\") " pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-sys\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-tuned\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.230539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190269 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-socket-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-run-k8s-cni-cncf-io\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-var-lib-cni-bin\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-daemon-config\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ngzn\" (UniqueName: \"kubernetes.io/projected/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-kube-api-access-2ngzn\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-tmp\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-hostroot\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-var-lib-openvswitch\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-env-overrides\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdrtx\" (UniqueName: \"kubernetes.io/projected/3b0b3fa4-bc82-4499-a622-3fb464453a1c-kube-api-access-pdrtx\") pod \"iptables-alerter-gf2wn\" (UID: \"3b0b3fa4-bc82-4499-a622-3fb464453a1c\") " pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d93eebd-7f1a-48fd-a0e3-51fe4ab46250-hosts-file\") pod \"node-resolver-ndr58\" (UID: \"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250\") " pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-lib-modules\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190634 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d93eebd-7f1a-48fd-a0e3-51fe4ab46250-hosts-file\") pod \"node-resolver-ndr58\" (UID: \"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250\") " pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-kubelet\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-run-systemd\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-sysctl-conf\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.231178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9412ed9-53f3-4469-99aa-119dab079331-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-slash\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190855 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b0b3fa4-bc82-4499-a622-3fb464453a1c-host-slash\") pod \"iptables-alerter-gf2wn\" (UID: \"3b0b3fa4-bc82-4499-a622-3fb464453a1c\") " pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-device-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-kubernetes\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9412ed9-53f3-4469-99aa-119dab079331-cni-binary-copy\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-run-ovn\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.190988 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-device-dir\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.191000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-node-log\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.194605 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0-agent-certs\") pod \"konnectivity-agent-4vndv\" (UID: \"1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0\") " pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.194718 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.194736 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.194745 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5z85n for pod openshift-network-diagnostics/network-check-target-j7pfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.194805 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n podName:60af647a-0dc7-4414-8cac-a61393b4cb51 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:06.694786009 +0000 UTC m=+2.993760964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5z85n" (UniqueName: "kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n") pod "network-check-target-j7pfq" (UID: "60af647a-0dc7-4414-8cac-a61393b4cb51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:06.234247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.200954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95n9j\" (UniqueName: \"kubernetes.io/projected/a1dcfc63-f4f1-4f3f-89cb-d646d00469d5-kube-api-access-95n9j\") pod \"aws-ebs-csi-driver-node-mxpl5\" (UID: \"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.235030 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.210621 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal" event={"ID":"6115ca8597103d232ce4b4815a5dee79","Type":"ContainerStarted","Data":"026801c1329b1eebca180e87c6e5a321eb80433ef6ddcd5db0b6510be2b7ca2c"} Jul 22 11:36:06.235030 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.212129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" event={"ID":"ab9df11ccd03f66e0259d1240fa3368b","Type":"ContainerStarted","Data":"d209f1e882234b0772fb27d76fb99c4ba6804f1a97a334779610de7370de0d6f"} Jul 22 11:36:06.291591 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b0b3fa4-bc82-4499-a622-3fb464453a1c-iptables-alerter-script\") pod \"iptables-alerter-gf2wn\" (UID: \"3b0b3fa4-bc82-4499-a622-3fb464453a1c\") " pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.291591 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-run\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.291591 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-system-cni-dir\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.291591 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-log-socket\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-sysconfig\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-host\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-cni-dir\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-var-lib-kubelet\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-cni-netd\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpt4g\" (UniqueName: \"kubernetes.io/projected/11037cad-c16a-4b04-b5ee-1287f5caef55-kube-api-access-dpt4g\") pod \"node-ca-mwg4w\" (UID: \"11037cad-c16a-4b04-b5ee-1287f5caef55\") " pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-cnibin\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-socket-dir-parent\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-etc-kubernetes\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-sysctl-d\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-systemd\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-os-release\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/787909e2-590a-44e5-baa0-d5d7b711aa6b-cni-binary-copy\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-cni-bin\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.291886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsn98\" (UniqueName: \"kubernetes.io/projected/3d93eebd-7f1a-48fd-a0e3-51fe4ab46250-kube-api-access-jsn98\") pod \"node-resolver-ndr58\" (UID: \"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250\") " pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-sys\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-tuned\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-run-k8s-cni-cncf-io\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-var-lib-cni-bin\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-daemon-config\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ngzn\" (UniqueName: \"kubernetes.io/projected/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-kube-api-access-2ngzn\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-tmp\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-hostroot\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-var-lib-openvswitch\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-env-overrides\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdrtx\" (UniqueName: \"kubernetes.io/projected/3b0b3fa4-bc82-4499-a622-3fb464453a1c-kube-api-access-pdrtx\") pod \"iptables-alerter-gf2wn\" (UID: \"3b0b3fa4-bc82-4499-a622-3fb464453a1c\") " pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-lib-modules\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-kubelet\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-run-systemd\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292166 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b0b3fa4-bc82-4499-a622-3fb464453a1c-iptables-alerter-script\") pod \"iptables-alerter-gf2wn\" (UID: \"3b0b3fa4-bc82-4499-a622-3fb464453a1c\") " pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-sysctl-conf\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.292640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9412ed9-53f3-4469-99aa-119dab079331-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-slash\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b0b3fa4-bc82-4499-a622-3fb464453a1c-host-slash\") pod \"iptables-alerter-gf2wn\" (UID: \"3b0b3fa4-bc82-4499-a622-3fb464453a1c\") " pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-kubernetes\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9412ed9-53f3-4469-99aa-119dab079331-cni-binary-copy\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-sysctl-conf\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-run-ovn\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-node-log\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1aca7ae8-e7fc-417d-840b-a7fa1155723f-kubelet-config\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-var-lib-cni-multus\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-ovn-node-metrics-cert\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-system-cni-dir\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-cnibin\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntn6k\" (UniqueName: \"kubernetes.io/projected/e9412ed9-53f3-4469-99aa-119dab079331-kube-api-access-ntn6k\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-systemd-units\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.293466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-run-netns\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-ovnkube-config\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-ovnkube-script-lib\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11037cad-c16a-4b04-b5ee-1287f5caef55-serviceca\") pod \"node-ca-mwg4w\" (UID: \"11037cad-c16a-4b04-b5ee-1287f5caef55\") " pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1aca7ae8-e7fc-417d-840b-a7fa1155723f-dbus\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11037cad-c16a-4b04-b5ee-1287f5caef55-host\") pod \"node-ca-mwg4w\" (UID: \"11037cad-c16a-4b04-b5ee-1287f5caef55\") " pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-os-release\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghzh\" (UniqueName: \"kubernetes.io/projected/787909e2-590a-44e5-baa0-d5d7b711aa6b-kube-api-access-tghzh\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-var-lib-kubelet\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wxs\" (UniqueName: \"kubernetes.io/projected/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-kube-api-access-p8wxs\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-system-cni-dir\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9412ed9-53f3-4469-99aa-119dab079331-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-conf-dir\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-run-multus-certs\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-modprobe-d\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.294263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9412ed9-53f3-4469-99aa-119dab079331-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlq8h\" (UniqueName: \"kubernetes.io/projected/64781c58-80c2-4b43-bc93-8bd1a7df80ca-kube-api-access-hlq8h\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-run-netns\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.292981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-etc-openvswitch\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-run-openvswitch\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-run-openvswitch\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-slash\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-log-socket\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-kubernetes\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b0b3fa4-bc82-4499-a622-3fb464453a1c-host-slash\") pod \"iptables-alerter-gf2wn\" (UID: \"3b0b3fa4-bc82-4499-a622-3fb464453a1c\") " pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-sysconfig\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-host\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-cni-dir\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-var-lib-kubelet\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293507 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-cni-netd\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9412ed9-53f3-4469-99aa-119dab079331-cni-binary-copy\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293690 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-run-ovn\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.295080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-cnibin\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293786 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1aca7ae8-e7fc-417d-840b-a7fa1155723f-kubelet-config\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.293819 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293835 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-socket-dir-parent\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.291737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-run\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.293876 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret podName:1aca7ae8-e7fc-417d-840b-a7fa1155723f nodeName:}" failed. No retries permitted until 2026-07-22 11:36:06.793860598 +0000 UTC m=+3.092835554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret") pod "global-pull-secret-syncer-gcrmm" (UID: "1aca7ae8-e7fc-417d-840b-a7fa1155723f") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-etc-kubernetes\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-var-lib-cni-multus\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293973 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-sysctl-d\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.294017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-systemd\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.294075 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-os-release\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.294154 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.294221 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs podName:64781c58-80c2-4b43-bc93-8bd1a7df80ca nodeName:}" failed. No retries permitted until 2026-07-22 11:36:06.794206051 +0000 UTC m=+3.093181001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs") pod "network-metrics-daemon-x92mg" (UID: "64781c58-80c2-4b43-bc93-8bd1a7df80ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.294378 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-systemd-units\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.294436 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-cni-bin\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.294660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-sys\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.294721 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/787909e2-590a-44e5-baa0-d5d7b711aa6b-cni-binary-copy\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.295886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.294787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-run-netns\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.295268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-ovnkube-config\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.295599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-ovnkube-script-lib\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.295833 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11037cad-c16a-4b04-b5ee-1287f5caef55-serviceca\") pod \"node-ca-mwg4w\" (UID: \"11037cad-c16a-4b04-b5ee-1287f5caef55\") " pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.295901 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1aca7ae8-e7fc-417d-840b-a7fa1155723f-dbus\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.295930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11037cad-c16a-4b04-b5ee-1287f5caef55-host\") pod \"node-ca-mwg4w\" (UID: \"11037cad-c16a-4b04-b5ee-1287f5caef55\") " pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.295974 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-os-release\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-run-netns\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-etc-openvswitch\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296236 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-var-lib-kubelet\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-var-lib-openvswitch\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296363 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-run-k8s-cni-cncf-io\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296404 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-var-lib-cni-bin\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-system-cni-dir\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.296703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296578 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-daemon-config\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.296985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9412ed9-53f3-4469-99aa-119dab079331-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.297037 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-multus-conf-dir\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.297078 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-host-run-multus-certs\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.293739 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-node-log\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.297134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-host-kubelet\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.297373 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/787909e2-590a-44e5-baa0-d5d7b711aa6b-hostroot\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.297384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-run-systemd\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.297416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9412ed9-53f3-4469-99aa-119dab079331-cnibin\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.297432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-lib-modules\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.297540 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-env-overrides\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.297755 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.297549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-modprobe-d\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.298875 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.298851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-ovn-node-metrics-cert\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.299787 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.299707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-tmp\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.300878 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.300721 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-etc-tuned\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.302601 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.302289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlq8h\" (UniqueName: \"kubernetes.io/projected/64781c58-80c2-4b43-bc93-8bd1a7df80ca-kube-api-access-hlq8h\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:06.302601 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.302498 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpt4g\" (UniqueName: \"kubernetes.io/projected/11037cad-c16a-4b04-b5ee-1287f5caef55-kube-api-access-dpt4g\") pod \"node-ca-mwg4w\" (UID: \"11037cad-c16a-4b04-b5ee-1287f5caef55\") " pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.304016 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.303992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsn98\" (UniqueName: \"kubernetes.io/projected/3d93eebd-7f1a-48fd-a0e3-51fe4ab46250-kube-api-access-jsn98\") pod \"node-resolver-ndr58\" (UID: \"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250\") " pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.308257 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.308217 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ngzn\" (UniqueName: \"kubernetes.io/projected/51efd00e-bce7-47b2-92ee-94fcd52ae3ab-kube-api-access-2ngzn\") pod \"ovnkube-node-5sctd\" (UID: \"51efd00e-bce7-47b2-92ee-94fcd52ae3ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.308669 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.308648 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghzh\" (UniqueName: \"kubernetes.io/projected/787909e2-590a-44e5-baa0-d5d7b711aa6b-kube-api-access-tghzh\") pod \"multus-lmkbc\" (UID: \"787909e2-590a-44e5-baa0-d5d7b711aa6b\") " pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.309284 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.309263 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wxs\" (UniqueName: \"kubernetes.io/projected/df03ccdb-d98c-4df0-884a-d674e4bd3dcd-kube-api-access-p8wxs\") pod \"tuned-42rnz\" (UID: \"df03ccdb-d98c-4df0-884a-d674e4bd3dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.310740 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.310708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdrtx\" (UniqueName: \"kubernetes.io/projected/3b0b3fa4-bc82-4499-a622-3fb464453a1c-kube-api-access-pdrtx\") pod \"iptables-alerter-gf2wn\" (UID: \"3b0b3fa4-bc82-4499-a622-3fb464453a1c\") " pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.310819 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.310781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntn6k\" (UniqueName: \"kubernetes.io/projected/e9412ed9-53f3-4469-99aa-119dab079331-kube-api-access-ntn6k\") pod \"multus-additional-cni-plugins-dbflp\" (UID: \"e9412ed9-53f3-4469-99aa-119dab079331\") " pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.324722 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.324702 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 11:36:06.395704 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.395669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:06.398576 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.398461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" Jul 22 11:36:06.408969 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:06.408938 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df4d1a6_bc29_4396_8ed8_a0e40c91c0f0.slice/crio-78c81fdd9d805f14d66b6cedfa2ee8d4f69f9142ca36bb1ba86ba408f32732ba WatchSource:0}: Error finding container 78c81fdd9d805f14d66b6cedfa2ee8d4f69f9142ca36bb1ba86ba408f32732ba: Status 404 returned error can't find the container with id 78c81fdd9d805f14d66b6cedfa2ee8d4f69f9142ca36bb1ba86ba408f32732ba Jul 22 11:36:06.412189 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:06.411890 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1dcfc63_f4f1_4f3f_89cb_d646d00469d5.slice/crio-51ce8c6130fc9788e06df7d38d254493e8354f8bfe41d41707f3ea7e29c96e1f WatchSource:0}: Error finding container 51ce8c6130fc9788e06df7d38d254493e8354f8bfe41d41707f3ea7e29c96e1f: Status 404 returned error can't find the container with id 51ce8c6130fc9788e06df7d38d254493e8354f8bfe41d41707f3ea7e29c96e1f Jul 22 11:36:06.412189 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.411998 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-42rnz" Jul 22 11:36:06.416637 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.416606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dbflp" Jul 22 11:36:06.426784 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.426766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ndr58" Jul 22 11:36:06.429890 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:06.429833 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9412ed9_53f3_4469_99aa_119dab079331.slice/crio-c7007138e4c74cfcd00afda76f75d4031432a8a3b293a5ed170096ed24de387a WatchSource:0}: Error finding container c7007138e4c74cfcd00afda76f75d4031432a8a3b293a5ed170096ed24de387a: Status 404 returned error can't find the container with id c7007138e4c74cfcd00afda76f75d4031432a8a3b293a5ed170096ed24de387a Jul 22 11:36:06.434874 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.434855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:06.438981 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:06.438916 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d93eebd_7f1a_48fd_a0e3_51fe4ab46250.slice/crio-532432a32a616b23cb9e51dd52c049a9932763df3746e37890db904ee9ba640c WatchSource:0}: Error finding container 532432a32a616b23cb9e51dd52c049a9932763df3746e37890db904ee9ba640c: Status 404 returned error can't find the container with id 532432a32a616b23cb9e51dd52c049a9932763df3746e37890db904ee9ba640c Jul 22 11:36:06.448082 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:06.447938 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51efd00e_bce7_47b2_92ee_94fcd52ae3ab.slice/crio-529746e7aff03c8db1b596548e0cb4c38df3f9b957dfec918faca55e5e879baa WatchSource:0}: Error finding container 529746e7aff03c8db1b596548e0cb4c38df3f9b957dfec918faca55e5e879baa: Status 404 returned error can't find the container with id 529746e7aff03c8db1b596548e0cb4c38df3f9b957dfec918faca55e5e879baa Jul 22 11:36:06.453568 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.453440 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gf2wn" Jul 22 11:36:06.463506 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.463487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mwg4w" Jul 22 11:36:06.465572 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:06.465549 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0b3fa4_bc82_4499_a622_3fb464453a1c.slice/crio-d77ceb85cc909e1f83d96c7435a95554f8ff57cded1792276667d43dbfba7a5b WatchSource:0}: Error finding container d77ceb85cc909e1f83d96c7435a95554f8ff57cded1792276667d43dbfba7a5b: Status 404 returned error can't find the container with id d77ceb85cc909e1f83d96c7435a95554f8ff57cded1792276667d43dbfba7a5b Jul 22 11:36:06.472431 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.472410 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lmkbc" Jul 22 11:36:06.474620 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:06.474494 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11037cad_c16a_4b04_b5ee_1287f5caef55.slice/crio-85d030a220449753a61ede2321a20ae5bc4590917287e08152c1db1a80ee2065 WatchSource:0}: Error finding container 85d030a220449753a61ede2321a20ae5bc4590917287e08152c1db1a80ee2065: Status 404 returned error can't find the container with id 85d030a220449753a61ede2321a20ae5bc4590917287e08152c1db1a80ee2065 Jul 22 11:36:06.481235 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:06.481210 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787909e2_590a_44e5_baa0_d5d7b711aa6b.slice/crio-d676a8e9fab49ab3298a85b8b0de37d38dbf0bb016ee4c396c519468cc03d7e4 WatchSource:0}: Error finding container d676a8e9fab49ab3298a85b8b0de37d38dbf0bb016ee4c396c519468cc03d7e4: Status 404 returned error can't find the container with id d676a8e9fab49ab3298a85b8b0de37d38dbf0bb016ee4c396c519468cc03d7e4 Jul 22 11:36:06.695628 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.695591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z85n\" (UniqueName: \"kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n\") pod \"network-check-target-j7pfq\" (UID: \"60af647a-0dc7-4414-8cac-a61393b4cb51\") " pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:06.695796 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.695768 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:36:06.695796 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.695790 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:36:06.695898 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.695800 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5z85n for pod openshift-network-diagnostics/network-check-target-j7pfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:06.695898 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.695854 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n podName:60af647a-0dc7-4414-8cac-a61393b4cb51 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:07.695840823 +0000 UTC m=+3.994815759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5z85n" (UniqueName: "kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n") pod "network-check-target-j7pfq" (UID: "60af647a-0dc7-4414-8cac-a61393b4cb51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:06.796531 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.796495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:06.796747 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:06.796561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:06.796747 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.796628 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:06.796747 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.796689 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs podName:64781c58-80c2-4b43-bc93-8bd1a7df80ca nodeName:}" failed. No retries permitted until 2026-07-22 11:36:07.796669791 +0000 UTC m=+4.095644728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs") pod "network-metrics-daemon-x92mg" (UID: "64781c58-80c2-4b43-bc93-8bd1a7df80ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:06.796747 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.796709 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:06.796937 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:06.796784 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret podName:1aca7ae8-e7fc-417d-840b-a7fa1155723f nodeName:}" failed. No retries permitted until 2026-07-22 11:36:07.796764514 +0000 UTC m=+4.095739453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret") pod "global-pull-secret-syncer-gcrmm" (UID: "1aca7ae8-e7fc-417d-840b-a7fa1155723f") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:07.133595 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.133441 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 11:31:06 +0000 UTC" deadline="2027-12-25 00:32:19.461620987 +0000 UTC" Jul 22 11:36:07.133595 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.133499 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12492h56m12.328125327s" Jul 22 11:36:07.216255 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.216216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lmkbc" event={"ID":"787909e2-590a-44e5-baa0-d5d7b711aa6b","Type":"ContainerStarted","Data":"d676a8e9fab49ab3298a85b8b0de37d38dbf0bb016ee4c396c519468cc03d7e4"} Jul 22 11:36:07.217632 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.217602 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mwg4w" event={"ID":"11037cad-c16a-4b04-b5ee-1287f5caef55","Type":"ContainerStarted","Data":"85d030a220449753a61ede2321a20ae5bc4590917287e08152c1db1a80ee2065"} Jul 22 11:36:07.219187 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.219153 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" event={"ID":"51efd00e-bce7-47b2-92ee-94fcd52ae3ab","Type":"ContainerStarted","Data":"529746e7aff03c8db1b596548e0cb4c38df3f9b957dfec918faca55e5e879baa"} Jul 22 11:36:07.220663 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.220642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" event={"ID":"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5","Type":"ContainerStarted","Data":"51ce8c6130fc9788e06df7d38d254493e8354f8bfe41d41707f3ea7e29c96e1f"} Jul 22 11:36:07.221900 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.221874 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4vndv" event={"ID":"1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0","Type":"ContainerStarted","Data":"78c81fdd9d805f14d66b6cedfa2ee8d4f69f9142ca36bb1ba86ba408f32732ba"} Jul 22 11:36:07.223423 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.223387 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gf2wn" event={"ID":"3b0b3fa4-bc82-4499-a622-3fb464453a1c","Type":"ContainerStarted","Data":"d77ceb85cc909e1f83d96c7435a95554f8ff57cded1792276667d43dbfba7a5b"} Jul 22 11:36:07.225180 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.225156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ndr58" event={"ID":"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250","Type":"ContainerStarted","Data":"532432a32a616b23cb9e51dd52c049a9932763df3746e37890db904ee9ba640c"} Jul 22 11:36:07.226539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.226512 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbflp" event={"ID":"e9412ed9-53f3-4469-99aa-119dab079331","Type":"ContainerStarted","Data":"c7007138e4c74cfcd00afda76f75d4031432a8a3b293a5ed170096ed24de387a"} Jul 22 11:36:07.227830 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.227807 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-42rnz" event={"ID":"df03ccdb-d98c-4df0-884a-d674e4bd3dcd","Type":"ContainerStarted","Data":"35e2a243e75d12376f34ade00df79404ee8b6580802431a9ef05480a7c367dec"} Jul 22 11:36:07.704806 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.704769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z85n\" (UniqueName: \"kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n\") pod \"network-check-target-j7pfq\" (UID: \"60af647a-0dc7-4414-8cac-a61393b4cb51\") " pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:07.705272 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:07.704941 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:36:07.705272 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:07.704960 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:36:07.705272 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:07.704972 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5z85n for pod openshift-network-diagnostics/network-check-target-j7pfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:07.705272 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:07.705033 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n podName:60af647a-0dc7-4414-8cac-a61393b4cb51 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:09.705014704 +0000 UTC m=+6.003989645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5z85n" (UniqueName: "kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n") pod "network-check-target-j7pfq" (UID: "60af647a-0dc7-4414-8cac-a61393b4cb51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:07.806204 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.805840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:07.806204 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:07.805922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:07.806204 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:07.806047 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:07.806204 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:07.806136 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs podName:64781c58-80c2-4b43-bc93-8bd1a7df80ca nodeName:}" failed. No retries permitted until 2026-07-22 11:36:09.806116367 +0000 UTC m=+6.105091317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs") pod "network-metrics-daemon-x92mg" (UID: "64781c58-80c2-4b43-bc93-8bd1a7df80ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:07.806726 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:07.806566 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:07.806726 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:07.806655 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret podName:1aca7ae8-e7fc-417d-840b-a7fa1155723f nodeName:}" failed. No retries permitted until 2026-07-22 11:36:09.806639709 +0000 UTC m=+6.105614648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret") pod "global-pull-secret-syncer-gcrmm" (UID: "1aca7ae8-e7fc-417d-840b-a7fa1155723f") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:08.137373 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:08.137282 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 11:31:06 +0000 UTC" deadline="2027-12-18 20:11:12.86230905 +0000 UTC" Jul 22 11:36:08.137373 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:08.137325 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12344h35m4.724988967s" Jul 22 11:36:08.201342 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:08.201314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:08.201342 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:08.201331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:08.201549 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:08.201503 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:08.201609 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:08.201569 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:08.201666 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:08.201625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:08.201731 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:08.201693 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:09.719655 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:09.719621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z85n\" (UniqueName: \"kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n\") pod \"network-check-target-j7pfq\" (UID: \"60af647a-0dc7-4414-8cac-a61393b4cb51\") " pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:09.720315 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:09.719773 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:36:09.720315 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:09.719792 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:36:09.720315 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:09.719804 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5z85n for pod openshift-network-diagnostics/network-check-target-j7pfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:09.720315 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:09.719865 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n podName:60af647a-0dc7-4414-8cac-a61393b4cb51 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:13.719845934 +0000 UTC m=+10.018820887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5z85n" (UniqueName: "kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n") pod "network-check-target-j7pfq" (UID: "60af647a-0dc7-4414-8cac-a61393b4cb51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:09.820132 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:09.820096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:09.820308 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:09.820181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:09.820308 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:09.820262 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:09.820414 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:09.820307 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:09.820414 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:09.820341 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret podName:1aca7ae8-e7fc-417d-840b-a7fa1155723f nodeName:}" failed. No retries permitted until 2026-07-22 11:36:13.820318555 +0000 UTC m=+10.119293491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret") pod "global-pull-secret-syncer-gcrmm" (UID: "1aca7ae8-e7fc-417d-840b-a7fa1155723f") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:09.820414 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:09.820362 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs podName:64781c58-80c2-4b43-bc93-8bd1a7df80ca nodeName:}" failed. No retries permitted until 2026-07-22 11:36:13.820352036 +0000 UTC m=+10.119326975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs") pod "network-metrics-daemon-x92mg" (UID: "64781c58-80c2-4b43-bc93-8bd1a7df80ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:10.201547 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:10.201515 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:10.201702 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:10.201638 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:10.201996 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:10.201970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:10.202096 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:10.202077 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:10.202305 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:10.202166 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:10.202305 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:10.202276 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:11.245390 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:11.245326 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" event={"ID":"ab9df11ccd03f66e0259d1240fa3368b","Type":"ContainerStarted","Data":"85ee0724ea87da5e73fe625ceaaef15434dc939ea7048512338bff58bfbe2f92"} Jul 22 11:36:12.201035 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:12.200964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:12.201035 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:12.201013 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:12.201305 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:12.200964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:12.201305 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:12.201107 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:12.201305 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:12.201213 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:12.201305 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:12.201289 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:13.750907 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:13.750810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z85n\" (UniqueName: \"kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n\") pod \"network-check-target-j7pfq\" (UID: \"60af647a-0dc7-4414-8cac-a61393b4cb51\") " pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:13.751230 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:13.750972 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:36:13.751230 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:13.750991 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:36:13.751230 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:13.751010 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5z85n for pod openshift-network-diagnostics/network-check-target-j7pfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:13.751230 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:13.751069 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n podName:60af647a-0dc7-4414-8cac-a61393b4cb51 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:21.751045759 +0000 UTC m=+18.050020708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5z85n" (UniqueName: "kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n") pod "network-check-target-j7pfq" (UID: "60af647a-0dc7-4414-8cac-a61393b4cb51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:13.852036 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:13.852011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:13.852184 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:13.852071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:13.852184 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:13.852171 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:13.852325 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:13.852250 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs podName:64781c58-80c2-4b43-bc93-8bd1a7df80ca nodeName:}" failed. No retries permitted until 2026-07-22 11:36:21.852230856 +0000 UTC m=+18.151205797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs") pod "network-metrics-daemon-x92mg" (UID: "64781c58-80c2-4b43-bc93-8bd1a7df80ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:13.852325 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:13.852171 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:13.852417 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:13.852332 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret podName:1aca7ae8-e7fc-417d-840b-a7fa1155723f nodeName:}" failed. No retries permitted until 2026-07-22 11:36:21.852314169 +0000 UTC m=+18.151289118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret") pod "global-pull-secret-syncer-gcrmm" (UID: "1aca7ae8-e7fc-417d-840b-a7fa1155723f") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:14.203614 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:14.203581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:14.203791 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:14.203699 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:14.203791 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:14.203742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:14.203791 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:14.203759 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:14.203951 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:14.203822 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:14.203951 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:14.203888 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:14.253793 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:14.253758 2579 generic.go:356] "Generic (PLEG): container finished" podID="ab9df11ccd03f66e0259d1240fa3368b" containerID="85ee0724ea87da5e73fe625ceaaef15434dc939ea7048512338bff58bfbe2f92" exitCode=0 Jul 22 11:36:14.253960 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:14.253907 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" event={"ID":"ab9df11ccd03f66e0259d1240fa3368b","Type":"ContainerDied","Data":"85ee0724ea87da5e73fe625ceaaef15434dc939ea7048512338bff58bfbe2f92"} Jul 22 11:36:14.255826 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:14.255800 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal" event={"ID":"6115ca8597103d232ce4b4815a5dee79","Type":"ContainerStarted","Data":"4e3d2aa0aaa696ffcd2b688f3d80817f5061269e9f4d3a502f6c0be6dad27500"} Jul 22 11:36:16.200980 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:16.200554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:16.200980 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:16.200611 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:16.200980 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:16.200560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:16.200980 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:16.200686 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:16.200980 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:16.200896 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:16.201516 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:16.200995 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:18.200509 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:18.200474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:18.200509 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:18.200503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:18.201012 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:18.200474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:18.201012 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:18.200605 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:18.201012 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:18.200713 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:18.201012 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:18.200799 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:20.200528 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:20.200491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:20.200528 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:20.200512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:20.200999 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:20.200624 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:20.200999 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:20.200748 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:20.200999 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:20.200783 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:20.200999 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:20.200863 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:21.818479 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:21.818445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z85n\" (UniqueName: \"kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n\") pod \"network-check-target-j7pfq\" (UID: \"60af647a-0dc7-4414-8cac-a61393b4cb51\") " pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:21.818983 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:21.818575 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:36:21.818983 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:21.818589 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:36:21.818983 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:21.818597 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5z85n for pod openshift-network-diagnostics/network-check-target-j7pfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:21.818983 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:21.818642 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n podName:60af647a-0dc7-4414-8cac-a61393b4cb51 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:37.818629045 +0000 UTC m=+34.117603980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5z85n" (UniqueName: "kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n") pod "network-check-target-j7pfq" (UID: "60af647a-0dc7-4414-8cac-a61393b4cb51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:21.918735 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:21.918705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:21.918890 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:21.918748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:21.918890 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:21.918845 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:21.918890 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:21.918859 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:21.919007 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:21.918902 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret podName:1aca7ae8-e7fc-417d-840b-a7fa1155723f nodeName:}" failed. No retries permitted until 2026-07-22 11:36:37.918884783 +0000 UTC m=+34.217859738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret") pod "global-pull-secret-syncer-gcrmm" (UID: "1aca7ae8-e7fc-417d-840b-a7fa1155723f") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:21.919007 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:21.918919 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs podName:64781c58-80c2-4b43-bc93-8bd1a7df80ca nodeName:}" failed. No retries permitted until 2026-07-22 11:36:37.918909792 +0000 UTC m=+34.217884727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs") pod "network-metrics-daemon-x92mg" (UID: "64781c58-80c2-4b43-bc93-8bd1a7df80ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:22.200959 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:22.200925 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:22.201142 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:22.200981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:22.201142 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:22.200925 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:22.201142 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:22.201054 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:22.201142 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:22.201132 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:22.201346 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:22.201233 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:24.201972 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:24.201942 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:24.202465 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:24.202045 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:24.202465 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:24.202135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:24.202465 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:24.202278 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:24.202465 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:24.202334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:24.202465 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:24.202423 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:25.279269 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.278918 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mwg4w" event={"ID":"11037cad-c16a-4b04-b5ee-1287f5caef55","Type":"ContainerStarted","Data":"1fe50a50126f35f46f5c832ac03056f25d50045240069b9a42b870e8c2c0514f"} Jul 22 11:36:25.288661 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.288356 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" event={"ID":"51efd00e-bce7-47b2-92ee-94fcd52ae3ab","Type":"ContainerStarted","Data":"60f1b9123d833321ead442758209bcbbfb1ad228b7bb643d1e746e659c8d1f40"} Jul 22 11:36:25.288661 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.288466 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" event={"ID":"51efd00e-bce7-47b2-92ee-94fcd52ae3ab","Type":"ContainerStarted","Data":"96f47b746b30627a22f329d185b0759974a934f28e776c88fa36f269c410a258"} Jul 22 11:36:25.288661 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.288518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" event={"ID":"51efd00e-bce7-47b2-92ee-94fcd52ae3ab","Type":"ContainerStarted","Data":"daa6c2220d378538b1e75aa691645272b61e2233a41a15f8d2d11537f2f6ae1f"} Jul 22 11:36:25.288661 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.288563 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" event={"ID":"51efd00e-bce7-47b2-92ee-94fcd52ae3ab","Type":"ContainerStarted","Data":"632215fc40e7de7c16a3162b2dff32a8dbf9aa532daef1f3e4da9f700dbbe2e0"} Jul 22 11:36:25.288661 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.288610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" event={"ID":"51efd00e-bce7-47b2-92ee-94fcd52ae3ab","Type":"ContainerStarted","Data":"5c8dbaca7a3e7207efd9784ff9b67a2105877d0d1131416800fea1c4ef72699a"} Jul 22 11:36:25.290904 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.290880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" event={"ID":"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5","Type":"ContainerStarted","Data":"e1acb1ddad9c11bd1b6d2ed0f29eaeec37aa9a4715847b73781ea44221a6acef"} Jul 22 11:36:25.293342 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.292904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4vndv" event={"ID":"1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0","Type":"ContainerStarted","Data":"e1d55c4005293436f6c3e6b60442fc8dd946ffd14739b81244d4f5b72d824578"} Jul 22 11:36:25.294928 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.294881 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-210.ec2.internal" podStartSLOduration=20.294866886 podStartE2EDuration="20.294866886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:14.286220716 +0000 UTC m=+10.585195725" watchObservedRunningTime="2026-07-22 11:36:25.294866886 +0000 UTC m=+21.593841844" Jul 22 11:36:25.296423 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.296223 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" event={"ID":"ab9df11ccd03f66e0259d1240fa3368b","Type":"ContainerStarted","Data":"19b0ae00edc96623f0ab5e79ce1837e412a4f7ac601b1afc82d08a182bda6716"} Jul 22 11:36:25.298401 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.298276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ndr58" event={"ID":"3d93eebd-7f1a-48fd-a0e3-51fe4ab46250","Type":"ContainerStarted","Data":"604ecd2623481426ce0573586d9db037fc2cc397c9a8fc0b34f1b834109e3df5"} Jul 22 11:36:25.301150 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.300939 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9412ed9-53f3-4469-99aa-119dab079331" containerID="879c69fb0c7c1bbb2b975ff870d68aa062bc2f9250542f876def07b753dece13" exitCode=0 Jul 22 11:36:25.301150 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.301003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbflp" event={"ID":"e9412ed9-53f3-4469-99aa-119dab079331","Type":"ContainerDied","Data":"879c69fb0c7c1bbb2b975ff870d68aa062bc2f9250542f876def07b753dece13"} Jul 22 11:36:25.303957 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.303932 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-42rnz" event={"ID":"df03ccdb-d98c-4df0-884a-d674e4bd3dcd","Type":"ContainerStarted","Data":"b734bd18c03ce8e9087cdd39822dd4e733dae29d8acf638cdad9ff5ad2a583c5"} Jul 22 11:36:25.311688 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.311644 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mwg4w" podStartSLOduration=7.251062313 podStartE2EDuration="21.311626659s" totalImagesPullingTime="14.060564346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:25.295657453 +0000 UTC m=+21.594632411" watchObservedRunningTime="2026-07-22 11:36:25.311626659 +0000 UTC m=+21.610601618" Jul 22 11:36:25.312006 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.311983 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4vndv" podStartSLOduration=7.184923027 podStartE2EDuration="21.311976749s" totalImagesPullingTime="14.127053722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:25.311285983 +0000 UTC m=+21.610260940" watchObservedRunningTime="2026-07-22 11:36:25.311976749 +0000 UTC m=+21.610951706" Jul 22 11:36:25.359991 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.359755 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-42rnz" podStartSLOduration=3.334437832 podStartE2EDuration="21.3597352s" totalImagesPullingTime="18.025297368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:25.359162926 +0000 UTC m=+21.658137907" watchObservedRunningTime="2026-07-22 11:36:25.3597352 +0000 UTC m=+21.658710156" Jul 22 11:36:25.378448 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.378376 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ndr58" podStartSLOduration=3.366116054 podStartE2EDuration="21.378354656s" totalImagesPullingTime="18.012238602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:25.377553022 +0000 UTC m=+21.676527980" watchObservedRunningTime="2026-07-22 11:36:25.378354656 +0000 UTC m=+21.677329614" Jul 22 11:36:25.399987 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:25.399920 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-210.ec2.internal" podStartSLOduration=20.399898532 podStartE2EDuration="20.399898532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:25.399788275 +0000 UTC m=+21.698763233" watchObservedRunningTime="2026-07-22 11:36:25.399898532 +0000 UTC m=+21.698873539" Jul 22 11:36:26.104852 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.104825 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 11:36:26.200492 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.200461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:26.200678 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.200461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:26.200678 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.200582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:26.200678 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:26.200581 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:26.200678 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:26.200658 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:26.200856 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.200703 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T11:36:26.104845764Z","UUID":"1e6e1bed-edb9-4bb8-bc07-41aaf7e8f640","Handler":null,"Name":"","Endpoint":""} Jul 22 11:36:26.200856 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:26.200760 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:26.202591 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.202568 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 11:36:26.202591 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.202602 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 11:36:26.307378 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.307261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lmkbc" event={"ID":"787909e2-590a-44e5-baa0-d5d7b711aa6b","Type":"ContainerStarted","Data":"f2812151bab55d8652855a17f4059274d0738c8d05d47004ebde66366113e63c"} Jul 22 11:36:26.311155 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.311123 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" event={"ID":"51efd00e-bce7-47b2-92ee-94fcd52ae3ab","Type":"ContainerStarted","Data":"f849ee1994dab2dc059adb1bf86262b0bef8f67f737247ee600d4bd155806865"} Jul 22 11:36:26.312993 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.312969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" event={"ID":"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5","Type":"ContainerStarted","Data":"7734e88fe6fd1b4bd8a1d7f61c5d0433e33e15a4dea4c504772dab443d0082a3"} Jul 22 11:36:26.314456 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.314429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gf2wn" event={"ID":"3b0b3fa4-bc82-4499-a622-3fb464453a1c","Type":"ContainerStarted","Data":"bedd9d12ff75aada0c3b9a6ef0e342c247c73b14e43189d0918de7fa00343db4"} Jul 22 11:36:26.326312 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.326270 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lmkbc" podStartSLOduration=3.563244938 podStartE2EDuration="22.326258187s" totalImagesPullingTime="18.763013249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:26.32578568 +0000 UTC m=+22.624760637" watchObservedRunningTime="2026-07-22 11:36:26.326258187 +0000 UTC m=+22.625233143" Jul 22 11:36:26.342345 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:26.342309 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gf2wn" podStartSLOduration=4.35957136 podStartE2EDuration="22.342295887s" totalImagesPullingTime="17.982724527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:26.341594151 +0000 UTC m=+22.640569108" watchObservedRunningTime="2026-07-22 11:36:26.342295887 +0000 UTC m=+22.641270843" Jul 22 11:36:28.200944 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:28.200914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:28.201501 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:28.200918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:28.201501 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:28.201027 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:28.201501 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:28.201042 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:28.201501 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:28.201148 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:28.201501 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:28.201219 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:28.262341 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:28.262315 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:28.262997 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:28.262971 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:28.322371 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:28.322341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" event={"ID":"51efd00e-bce7-47b2-92ee-94fcd52ae3ab","Type":"ContainerStarted","Data":"367070f2f03adcb91795647dcb7e1c283ffab61b62b8d54777efff8dd18f5739"} Jul 22 11:36:28.324474 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:28.324450 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" event={"ID":"a1dcfc63-f4f1-4f3f-89cb-d646d00469d5","Type":"ContainerStarted","Data":"106f3ec780a95314d347c901423663c7feeca1aee45fb77651b814a48f63a0cc"} Jul 22 11:36:28.345207 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:28.345150 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxpl5" podStartSLOduration=3.761906482 podStartE2EDuration="24.345133989s" totalImagesPullingTime="20.583227507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:28.344802096 +0000 UTC m=+24.643777053" watchObservedRunningTime="2026-07-22 11:36:28.345133989 +0000 UTC m=+24.644108947" Jul 22 11:36:28.963098 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:28.963056 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:28.963792 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:28.963659 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4vndv" Jul 22 11:36:30.200805 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:30.200593 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:30.201665 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:30.200596 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:30.201665 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:30.200827 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:30.201665 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:30.200671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:30.201665 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:30.200908 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:30.201665 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:30.200983 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:30.329824 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:30.329802 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9412ed9-53f3-4469-99aa-119dab079331" containerID="8436e972d8b1460ca5803afc1961a400d8fa60581d89f5e41d938bef5efc147e" exitCode=0 Jul 22 11:36:30.329965 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:30.329830 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbflp" event={"ID":"e9412ed9-53f3-4469-99aa-119dab079331","Type":"ContainerDied","Data":"8436e972d8b1460ca5803afc1961a400d8fa60581d89f5e41d938bef5efc147e"} Jul 22 11:36:30.333538 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:30.333431 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" event={"ID":"51efd00e-bce7-47b2-92ee-94fcd52ae3ab","Type":"ContainerStarted","Data":"12d638e0fdbd3395c6a85c7451631fa0ac888d57422849fcba432e0c015cac2a"} Jul 22 11:36:30.333706 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:30.333686 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:30.333820 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:30.333747 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:30.348860 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:30.348838 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:30.381995 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:30.381954 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" podStartSLOduration=8.320413776 podStartE2EDuration="26.381942762s" totalImagesPullingTime="18.061528986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:30.381661343 +0000 UTC m=+26.680636337" watchObservedRunningTime="2026-07-22 11:36:30.381942762 +0000 UTC m=+26.680917722" Jul 22 11:36:31.335846 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:31.335657 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:31.349822 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:31.349791 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:36:31.994756 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:31.994720 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j7pfq"] Jul 22 11:36:31.994912 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:31.994852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:31.994960 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:31.994938 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:31.997426 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:31.997394 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gcrmm"] Jul 22 11:36:31.997558 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:31.997489 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:31.997622 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:31.997577 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:31.998093 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:31.998066 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x92mg"] Jul 22 11:36:31.998243 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:31.998163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:31.998313 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:31.998264 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:32.339599 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:32.339521 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9412ed9-53f3-4469-99aa-119dab079331" containerID="ceb185d014499bddbc1cdeef6af8c0c232c839bb11ffb7e328789b1c5aab0cd6" exitCode=0 Jul 22 11:36:32.339991 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:32.339607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbflp" event={"ID":"e9412ed9-53f3-4469-99aa-119dab079331","Type":"ContainerDied","Data":"ceb185d014499bddbc1cdeef6af8c0c232c839bb11ffb7e328789b1c5aab0cd6"} Jul 22 11:36:33.201143 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:33.201114 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:33.201312 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:33.201250 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:34.201346 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:34.201319 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:34.201690 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:34.201394 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:34.201690 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:34.201475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:34.201690 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:34.201564 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:34.346565 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:34.346534 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9412ed9-53f3-4469-99aa-119dab079331" containerID="a729c15511219509250113b903c4956f5d14b9322c55b5525b0a27170bba5d99" exitCode=0 Jul 22 11:36:34.346745 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:34.346575 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbflp" event={"ID":"e9412ed9-53f3-4469-99aa-119dab079331","Type":"ContainerDied","Data":"a729c15511219509250113b903c4956f5d14b9322c55b5525b0a27170bba5d99"} Jul 22 11:36:35.200760 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:35.200730 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:35.200933 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:35.200835 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:36.200559 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:36.200522 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:36.201041 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:36.200658 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7pfq" podUID="60af647a-0dc7-4414-8cac-a61393b4cb51" Jul 22 11:36:36.201041 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:36.200715 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:36.201041 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:36.200835 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x92mg" podUID="64781c58-80c2-4b43-bc93-8bd1a7df80ca" Jul 22 11:36:37.200518 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:37.200482 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:37.200701 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:37.200607 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gcrmm" podUID="1aca7ae8-e7fc-417d-840b-a7fa1155723f" Jul 22 11:36:37.846277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:37.846244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z85n\" (UniqueName: \"kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n\") pod \"network-check-target-j7pfq\" (UID: \"60af647a-0dc7-4414-8cac-a61393b4cb51\") " pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:37.846449 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:37.846421 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 11:36:37.846449 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:37.846446 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 11:36:37.846541 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:37.846459 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5z85n for pod openshift-network-diagnostics/network-check-target-j7pfq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:37.846541 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:37.846520 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n podName:60af647a-0dc7-4414-8cac-a61393b4cb51 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:09.846501782 +0000 UTC m=+66.145476723 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5z85n" (UniqueName: "kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n") pod "network-check-target-j7pfq" (UID: "60af647a-0dc7-4414-8cac-a61393b4cb51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 11:36:37.947238 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:37.947204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:37.947424 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:37.947265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:37.947424 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:37.947365 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:37.947424 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:37.947417 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs podName:64781c58-80c2-4b43-bc93-8bd1a7df80ca nodeName:}" failed. No retries permitted until 2026-07-22 11:37:09.947400615 +0000 UTC m=+66.246375551 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs") pod "network-metrics-daemon-x92mg" (UID: "64781c58-80c2-4b43-bc93-8bd1a7df80ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 11:36:37.947581 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:37.947367 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:37.947581 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:37.947503 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret podName:1aca7ae8-e7fc-417d-840b-a7fa1155723f nodeName:}" failed. No retries permitted until 2026-07-22 11:37:09.947486986 +0000 UTC m=+66.246461950 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret") pod "global-pull-secret-syncer-gcrmm" (UID: "1aca7ae8-e7fc-417d-840b-a7fa1155723f") : object "kube-system"/"original-pull-secret" not registered Jul 22 11:36:38.035008 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.034976 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-210.ec2.internal" event="NodeReady" Jul 22 11:36:38.035209 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.035137 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 11:36:38.076036 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.075997 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f98d8b84f-sgdh6"] Jul 22 11:36:38.078432 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.078412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.083491 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.083465 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 11:36:38.083725 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.083704 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 11:36:38.083824 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.083753 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 11:36:38.084028 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.084009 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4jlt2\"" Jul 22 11:36:38.090263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.090245 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 11:36:38.096119 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.096098 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cp489"] Jul 22 11:36:38.097988 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.097971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.102049 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.102028 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 11:36:38.102135 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.102035 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dg84n\"" Jul 22 11:36:38.102135 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.102055 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 11:36:38.110125 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.110101 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f98d8b84f-sgdh6"] Jul 22 11:36:38.110731 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.110714 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r45q9"] Jul 22 11:36:38.112514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.112496 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r45q9" Jul 22 11:36:38.114653 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.114635 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cp489"] Jul 22 11:36:38.115998 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.115977 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 11:36:38.116103 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.115976 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 11:36:38.116103 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.115984 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 11:36:38.116103 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.116079 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7dvg\"" Jul 22 11:36:38.130306 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.130274 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r45q9"] Jul 22 11:36:38.204583 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.204511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:36:38.204583 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.204531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:36:38.207876 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.207853 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 11:36:38.208008 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.207960 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dd5dh\"" Jul 22 11:36:38.208127 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.208106 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 11:36:38.208254 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.208225 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 11:36:38.208332 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.208282 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4v9m8\"" Jul 22 11:36:38.249228 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-image-registry-private-configuration\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.249382 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-bound-sa-token\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.249382 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft8nf\" (UniqueName: \"kubernetes.io/projected/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-kube-api-access-ft8nf\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.249382 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ba37fbb-6fae-47d6-b308-b232339a177a-ca-trust-extracted\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.249382 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7879951c-c75e-42ed-a411-5cebada8a304-cert\") pod \"ingress-canary-r45q9\" (UID: \"7879951c-c75e-42ed-a411-5cebada8a304\") " pod="openshift-ingress-canary/ingress-canary-r45q9" Jul 22 11:36:38.249382 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjhv2\" (UniqueName: \"kubernetes.io/projected/7879951c-c75e-42ed-a411-5cebada8a304-kube-api-access-wjhv2\") pod \"ingress-canary-r45q9\" (UID: \"7879951c-c75e-42ed-a411-5cebada8a304\") " pod="openshift-ingress-canary/ingress-canary-r45q9" Jul 22 11:36:38.249382 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8lj\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-kube-api-access-4d8lj\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.249655 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-trusted-ca\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.249655 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-certificates\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.249655 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-installation-pull-secrets\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.249655 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.249655 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-tmp-dir\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.249655 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.249655 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.249620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-config-volume\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.350585 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.350381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-trusted-ca\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.350691 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.350625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-certificates\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.350752 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.350686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-installation-pull-secrets\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.350810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.350790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.350865 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.350838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-tmp-dir\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.350913 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.350864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.350913 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.350894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-config-volume\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.351006 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:38.350919 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:36:38.351006 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.350933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-image-registry-private-configuration\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.351006 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:38.350980 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls podName:869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:38.850960672 +0000 UTC m=+35.149935610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls") pod "dns-default-cp489" (UID: "869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2") : secret "dns-default-metrics-tls" not found Jul 22 11:36:38.351006 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.350998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-bound-sa-token\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.351247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.351032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8nf\" (UniqueName: \"kubernetes.io/projected/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-kube-api-access-ft8nf\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.351247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.351061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ba37fbb-6fae-47d6-b308-b232339a177a-ca-trust-extracted\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.351247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.351084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7879951c-c75e-42ed-a411-5cebada8a304-cert\") pod \"ingress-canary-r45q9\" (UID: \"7879951c-c75e-42ed-a411-5cebada8a304\") " pod="openshift-ingress-canary/ingress-canary-r45q9" Jul 22 11:36:38.351247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.351105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjhv2\" (UniqueName: \"kubernetes.io/projected/7879951c-c75e-42ed-a411-5cebada8a304-kube-api-access-wjhv2\") pod \"ingress-canary-r45q9\" (UID: \"7879951c-c75e-42ed-a411-5cebada8a304\") " pod="openshift-ingress-canary/ingress-canary-r45q9" Jul 22 11:36:38.351247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.351131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8lj\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-kube-api-access-4d8lj\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.351247 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.351183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-tmp-dir\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.351518 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:38.351249 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:36:38.351518 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:38.351275 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f98d8b84f-sgdh6: secret "image-registry-tls" not found Jul 22 11:36:38.351518 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.351285 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-certificates\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.351518 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:38.351328 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls podName:8ba37fbb-6fae-47d6-b308-b232339a177a nodeName:}" failed. No retries permitted until 2026-07-22 11:36:38.85131336 +0000 UTC m=+35.150288302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls") pod "image-registry-f98d8b84f-sgdh6" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a") : secret "image-registry-tls" not found Jul 22 11:36:38.351687 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.351608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-trusted-ca\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.351736 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.351717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ba37fbb-6fae-47d6-b308-b232339a177a-ca-trust-extracted\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.351785 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.351738 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-config-volume\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.355345 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.355321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7879951c-c75e-42ed-a411-5cebada8a304-cert\") pod \"ingress-canary-r45q9\" (UID: \"7879951c-c75e-42ed-a411-5cebada8a304\") " pod="openshift-ingress-canary/ingress-canary-r45q9" Jul 22 11:36:38.355468 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.355376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-image-registry-private-configuration\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.355468 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.355433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-installation-pull-secrets\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.360073 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.360042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbflp" event={"ID":"e9412ed9-53f3-4469-99aa-119dab079331","Type":"ContainerStarted","Data":"5be14d170521cced64840342623687814a4c0c1997ee518b2e9fa2596fd25f5f"} Jul 22 11:36:38.361035 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.361011 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8lj\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-kube-api-access-4d8lj\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.362007 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.361982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-bound-sa-token\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.362097 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.362077 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjhv2\" (UniqueName: \"kubernetes.io/projected/7879951c-c75e-42ed-a411-5cebada8a304-kube-api-access-wjhv2\") pod \"ingress-canary-r45q9\" (UID: \"7879951c-c75e-42ed-a411-5cebada8a304\") " pod="openshift-ingress-canary/ingress-canary-r45q9" Jul 22 11:36:38.362604 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.362585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft8nf\" (UniqueName: \"kubernetes.io/projected/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-kube-api-access-ft8nf\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.422923 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.422893 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r45q9" Jul 22 11:36:38.568821 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.568790 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r45q9"] Jul 22 11:36:38.572998 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:36:38.572975 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7879951c_c75e_42ed_a411_5cebada8a304.slice/crio-b89d25e0b53aba1ce8986f18f47c214cbcfe774d2e551b8e657f002f7ca4e13f WatchSource:0}: Error finding container b89d25e0b53aba1ce8986f18f47c214cbcfe774d2e551b8e657f002f7ca4e13f: Status 404 returned error can't find the container with id b89d25e0b53aba1ce8986f18f47c214cbcfe774d2e551b8e657f002f7ca4e13f Jul 22 11:36:38.854391 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.854290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:38.854391 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:38.854345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:38.854598 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:38.854445 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:36:38.854598 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:38.854479 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:36:38.854598 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:38.854493 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f98d8b84f-sgdh6: secret "image-registry-tls" not found Jul 22 11:36:38.854598 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:38.854528 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls podName:869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:39.854510493 +0000 UTC m=+36.153485428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls") pod "dns-default-cp489" (UID: "869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2") : secret "dns-default-metrics-tls" not found Jul 22 11:36:38.854598 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:38.854551 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls podName:8ba37fbb-6fae-47d6-b308-b232339a177a nodeName:}" failed. No retries permitted until 2026-07-22 11:36:39.854537569 +0000 UTC m=+36.153512522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls") pod "image-registry-f98d8b84f-sgdh6" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a") : secret "image-registry-tls" not found Jul 22 11:36:39.201282 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:39.201243 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:36:39.204413 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:39.204391 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 11:36:39.365467 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:39.365434 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9412ed9-53f3-4469-99aa-119dab079331" containerID="5be14d170521cced64840342623687814a4c0c1997ee518b2e9fa2596fd25f5f" exitCode=0 Jul 22 11:36:39.366223 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:39.365516 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbflp" event={"ID":"e9412ed9-53f3-4469-99aa-119dab079331","Type":"ContainerDied","Data":"5be14d170521cced64840342623687814a4c0c1997ee518b2e9fa2596fd25f5f"} Jul 22 11:36:39.366572 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:39.366531 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r45q9" event={"ID":"7879951c-c75e-42ed-a411-5cebada8a304","Type":"ContainerStarted","Data":"b89d25e0b53aba1ce8986f18f47c214cbcfe774d2e551b8e657f002f7ca4e13f"} Jul 22 11:36:39.862294 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:39.862209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:39.862294 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:39.862265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:39.862509 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:39.862368 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:36:39.862509 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:39.862430 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls podName:869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:41.862413205 +0000 UTC m=+38.161388144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls") pod "dns-default-cp489" (UID: "869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2") : secret "dns-default-metrics-tls" not found Jul 22 11:36:39.862509 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:39.862442 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:36:39.862509 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:39.862464 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f98d8b84f-sgdh6: secret "image-registry-tls" not found Jul 22 11:36:39.862723 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:39.862522 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls podName:8ba37fbb-6fae-47d6-b308-b232339a177a nodeName:}" failed. No retries permitted until 2026-07-22 11:36:41.862504822 +0000 UTC m=+38.161479775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls") pod "image-registry-f98d8b84f-sgdh6" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a") : secret "image-registry-tls" not found Jul 22 11:36:40.372381 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:40.372350 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9412ed9-53f3-4469-99aa-119dab079331" containerID="77b5c9ec462a5343d944848f657fc795b3df2d55f5f33ab1bdc28331aa7fe76c" exitCode=0 Jul 22 11:36:40.372792 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:40.372415 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbflp" event={"ID":"e9412ed9-53f3-4469-99aa-119dab079331","Type":"ContainerDied","Data":"77b5c9ec462a5343d944848f657fc795b3df2d55f5f33ab1bdc28331aa7fe76c"} Jul 22 11:36:41.377409 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.377367 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r45q9" event={"ID":"7879951c-c75e-42ed-a411-5cebada8a304","Type":"ContainerStarted","Data":"c59a8999ff87a42b2f392864152c73b151ba2252b52c889399725803954c9cd1"} Jul 22 11:36:41.380934 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.380905 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbflp" event={"ID":"e9412ed9-53f3-4469-99aa-119dab079331","Type":"ContainerStarted","Data":"f74437a308c3f3bd01cf8842c1d3f8840cae477633835903b09b8ad4b73f1b2f"} Jul 22 11:36:41.396005 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.395944 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r45q9" podStartSLOduration=0.740867254 podStartE2EDuration="3.395929187s" totalImagesPullingTime="2.655061933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:41.395538356 +0000 UTC m=+37.694513323" watchObservedRunningTime="2026-07-22 11:36:41.395929187 +0000 UTC m=+37.694904145" Jul 22 11:36:41.419364 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.419309 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dbflp" podStartSLOduration=9.048695005 podStartE2EDuration="37.419293098s" totalImagesPullingTime="28.370598093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:36:41.417974087 +0000 UTC m=+37.716949075" watchObservedRunningTime="2026-07-22 11:36:41.419293098 +0000 UTC m=+37.718268057" Jul 22 11:36:41.576658 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.576578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:41.661824 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.661793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:41.840847 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.840769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:41.876692 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.876662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:41.876817 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.876697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:41.876855 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:41.876811 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:36:41.876885 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:41.876860 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:36:41.876885 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:41.876870 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f98d8b84f-sgdh6: secret "image-registry-tls" not found Jul 22 11:36:41.876885 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:41.876874 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls podName:869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:45.876852812 +0000 UTC m=+42.175827751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls") pod "dns-default-cp489" (UID: "869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2") : secret "dns-default-metrics-tls" not found Jul 22 11:36:41.876974 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:41.876906 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls podName:8ba37fbb-6fae-47d6-b308-b232339a177a nodeName:}" failed. No retries permitted until 2026-07-22 11:36:45.876894211 +0000 UTC m=+42.175869175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls") pod "image-registry-f98d8b84f-sgdh6" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a") : secret "image-registry-tls" not found Jul 22 11:36:41.913060 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.913029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:41.975702 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:41.975677 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:42.050884 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:42.050852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:43.778522 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:43.778490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:45.454277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:45.454245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:45.518875 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:45.518845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:45.903737 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:45.903708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:45.903885 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:45.903747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:45.903885 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:45.903865 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:36:45.903991 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:45.903923 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls podName:869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2 nodeName:}" failed. No retries permitted until 2026-07-22 11:36:53.903908545 +0000 UTC m=+50.202883484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls") pod "dns-default-cp489" (UID: "869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2") : secret "dns-default-metrics-tls" not found Jul 22 11:36:45.903991 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:45.903868 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:36:45.903991 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:45.903943 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f98d8b84f-sgdh6: secret "image-registry-tls" not found Jul 22 11:36:45.903991 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:45.903979 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls podName:8ba37fbb-6fae-47d6-b308-b232339a177a nodeName:}" failed. No retries permitted until 2026-07-22 11:36:53.903970373 +0000 UTC m=+50.202945323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls") pod "image-registry-f98d8b84f-sgdh6" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a") : secret "image-registry-tls" not found Jul 22 11:36:47.841550 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:47.841518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:47.900617 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:47.900580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:36:53.959451 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:53.959414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:36:53.959828 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:36:53.959457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:36:53.959828 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:53.959561 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:36:53.959828 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:53.959563 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:36:53.959828 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:53.959627 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls podName:869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:09.959610363 +0000 UTC m=+66.258585298 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls") pod "dns-default-cp489" (UID: "869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2") : secret "dns-default-metrics-tls" not found Jul 22 11:36:53.959828 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:53.959572 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f98d8b84f-sgdh6: secret "image-registry-tls" not found Jul 22 11:36:53.959828 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:36:53.959694 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls podName:8ba37fbb-6fae-47d6-b308-b232339a177a nodeName:}" failed. No retries permitted until 2026-07-22 11:37:09.959682895 +0000 UTC m=+66.258657831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls") pod "image-registry-f98d8b84f-sgdh6" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a") : secret "image-registry-tls" not found Jul 22 11:37:03.363415 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:03.363385 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sctd" Jul 22 11:37:03.622393 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:03.622305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:37:03.724387 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:03.724347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" limit=1024 Jul 22 11:37:09.594945 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.594910 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws"] Jul 22 11:37:09.633351 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.633315 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws"] Jul 22 11:37:09.633351 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.633349 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-t5smg"] Jul 22 11:37:09.633531 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.633441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws" Jul 22 11:37:09.636589 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.636565 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 22 11:37:09.636725 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.636565 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:37:09.636725 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.636565 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cpjtp\"" Jul 22 11:37:09.650873 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.650852 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6"] Jul 22 11:37:09.651010 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.650967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.653843 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.653825 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 22 11:37:09.653944 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.653872 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-plt24\"" Jul 22 11:37:09.654088 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.654067 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 22 11:37:09.654214 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.654095 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 22 11:37:09.654214 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.654113 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:37:09.658685 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.658662 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 22 11:37:09.664919 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.664900 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6"] Jul 22 11:37:09.665001 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.664928 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-t5smg"] Jul 22 11:37:09.665001 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.664995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.667441 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.667422 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 22 11:37:09.667521 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.667450 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:37:09.667663 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.667650 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 22 11:37:09.667731 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.667699 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 22 11:37:09.667787 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.667766 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8kgbh\"" Jul 22 11:37:09.670450 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.670421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4ca318a-3b21-42da-9525-6e907b40970d-config\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.670516 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.670468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5n6\" (UniqueName: \"kubernetes.io/projected/a4ca318a-3b21-42da-9525-6e907b40970d-kube-api-access-6v5n6\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.670575 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.670538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvtk\" (UniqueName: \"kubernetes.io/projected/7a58d7a4-7171-4103-8e5f-1d921c7572af-kube-api-access-bmvtk\") pod \"volume-data-source-validator-78cf4c6f6-xtxws\" (UID: \"7a58d7a4-7171-4103-8e5f-1d921c7572af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws" Jul 22 11:37:09.670627 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.670593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4ca318a-3b21-42da-9525-6e907b40970d-trusted-ca\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.670675 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.670642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4ca318a-3b21-42da-9525-6e907b40970d-serving-cert\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.695770 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.695747 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt"] Jul 22 11:37:09.713339 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.713312 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt"] Jul 22 11:37:09.713420 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.713379 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt" Jul 22 11:37:09.715843 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.715823 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kzsvj"] Jul 22 11:37:09.716938 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.716923 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mfkbr\"" Jul 22 11:37:09.737316 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.737286 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-67fc875744-bt6br"] Jul 22 11:37:09.737441 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.737423 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.740051 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.740034 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 22 11:37:09.740341 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.740323 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 11:37:09.740438 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.740384 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 11:37:09.740438 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.740409 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 22 11:37:09.740585 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.740534 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-q5q6w\"" Jul 22 11:37:09.745443 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.745424 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 22 11:37:09.761072 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.761055 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8"] Jul 22 11:37:09.761187 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.761172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.763744 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.763727 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 22 11:37:09.763922 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.763904 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 22 11:37:09.764179 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.764162 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 22 11:37:09.764287 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.764150 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 11:37:09.764348 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.764315 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-klj97\"" Jul 22 11:37:09.764448 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.764432 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 22 11:37:09.764634 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.764620 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 11:37:09.771633 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvtk\" (UniqueName: \"kubernetes.io/projected/7a58d7a4-7171-4103-8e5f-1d921c7572af-kube-api-access-bmvtk\") pod \"volume-data-source-validator-78cf4c6f6-xtxws\" (UID: \"7a58d7a4-7171-4103-8e5f-1d921c7572af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws" Jul 22 11:37:09.771725 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4ca318a-3b21-42da-9525-6e907b40970d-trusted-ca\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.771725 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771692 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjjhd\" (UniqueName: \"kubernetes.io/projected/62aac9b7-2b61-47c3-b68f-0cde3dee4534-kube-api-access-wjjhd\") pod \"network-check-source-7bc5b6bf85-r64wt\" (UID: \"62aac9b7-2b61-47c3-b68f-0cde3dee4534\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt" Jul 22 11:37:09.771725 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d0b446-3750-4f25-b181-87b1b351863d-tmp\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.771873 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4ca318a-3b21-42da-9525-6e907b40970d-serving-cert\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.771873 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhx42\" (UniqueName: \"kubernetes.io/projected/41df392a-72e8-4b96-b6dd-a8eaf50b322b-kube-api-access-nhx42\") pod \"kube-storage-version-migrator-operator-7658d9b749-j5bf6\" (UID: \"41df392a-72e8-4b96-b6dd-a8eaf50b322b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.771873 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41df392a-72e8-4b96-b6dd-a8eaf50b322b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-j5bf6\" (UID: \"41df392a-72e8-4b96-b6dd-a8eaf50b322b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.771873 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d0b446-3750-4f25-b181-87b1b351863d-serving-cert\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.772027 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-stats-auth\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.772027 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d0b446-3750-4f25-b181-87b1b351863d-service-ca-bundle\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.772027 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.771990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d0b446-3750-4f25-b181-87b1b351863d-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.772130 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e0d0b446-3750-4f25-b181-87b1b351863d-snapshots\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.772130 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4ca318a-3b21-42da-9525-6e907b40970d-config\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.772130 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcch6\" (UniqueName: \"kubernetes.io/projected/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-kube-api-access-vcch6\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.772269 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5n6\" (UniqueName: \"kubernetes.io/projected/a4ca318a-3b21-42da-9525-6e907b40970d-kube-api-access-6v5n6\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.772269 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.772269 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.772269 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41df392a-72e8-4b96-b6dd-a8eaf50b322b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-j5bf6\" (UID: \"41df392a-72e8-4b96-b6dd-a8eaf50b322b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.772437 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-default-certificate\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.772437 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw97d\" (UniqueName: \"kubernetes.io/projected/e0d0b446-3750-4f25-b181-87b1b351863d-kube-api-access-fw97d\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.772503 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772447 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4ca318a-3b21-42da-9525-6e907b40970d-trusted-ca\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.772582 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.772565 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4ca318a-3b21-42da-9525-6e907b40970d-config\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.775440 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.775421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4ca318a-3b21-42da-9525-6e907b40970d-serving-cert\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.779619 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.779575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvtk\" (UniqueName: \"kubernetes.io/projected/7a58d7a4-7171-4103-8e5f-1d921c7572af-kube-api-access-bmvtk\") pod \"volume-data-source-validator-78cf4c6f6-xtxws\" (UID: \"7a58d7a4-7171-4103-8e5f-1d921c7572af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws" Jul 22 11:37:09.779760 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.779692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5n6\" (UniqueName: \"kubernetes.io/projected/a4ca318a-3b21-42da-9525-6e907b40970d-kube-api-access-6v5n6\") pod \"console-operator-565b7b569d-t5smg\" (UID: \"a4ca318a-3b21-42da-9525-6e907b40970d\") " pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.785266 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.785247 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8"] Jul 22 11:37:09.785343 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.785274 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kzsvj"] Jul 22 11:37:09.785343 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.785292 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67fc875744-bt6br"] Jul 22 11:37:09.785407 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.785345 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:09.787706 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.787686 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 22 11:37:09.787796 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.787750 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 11:37:09.787796 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.787760 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 11:37:09.787796 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.787781 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wtcxt\"" Jul 22 11:37:09.789063 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.789048 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 22 11:37:09.795587 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.795569 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c"] Jul 22 11:37:09.817382 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.817360 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m"] Jul 22 11:37:09.817503 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.817485 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:09.819845 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.819823 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 11:37:09.819967 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.819854 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9c5zk\"" Jul 22 11:37:09.820030 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.820009 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:37:09.820155 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.820139 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 11:37:09.841582 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.841560 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c"] Jul 22 11:37:09.841691 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.841592 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m"] Jul 22 11:37:09.841691 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.841669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:09.844088 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.844071 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 22 11:37:09.844210 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.844115 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 22 11:37:09.844210 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.844118 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 22 11:37:09.844210 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.844180 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m6rjs\"" Jul 22 11:37:09.844423 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.844409 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 22 11:37:09.872952 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.872927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhx42\" (UniqueName: \"kubernetes.io/projected/41df392a-72e8-4b96-b6dd-a8eaf50b322b-kube-api-access-nhx42\") pod \"kube-storage-version-migrator-operator-7658d9b749-j5bf6\" (UID: \"41df392a-72e8-4b96-b6dd-a8eaf50b322b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.873110 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.872972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41df392a-72e8-4b96-b6dd-a8eaf50b322b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-j5bf6\" (UID: \"41df392a-72e8-4b96-b6dd-a8eaf50b322b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.873110 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.872991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/96025fea-bf2d-43e3-9506-7ebea486627c-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:09.873110 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d0b446-3750-4f25-b181-87b1b351863d-serving-cert\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.873110 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-stats-auth\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.873110 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d0b446-3750-4f25-b181-87b1b351863d-service-ca-bundle\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.873110 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z85n\" (UniqueName: \"kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n\") pod \"network-check-target-j7pfq\" (UID: \"60af647a-0dc7-4414-8cac-a61393b4cb51\") " pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:37:09.873436 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d0b446-3750-4f25-b181-87b1b351863d-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.873436 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e0d0b446-3750-4f25-b181-87b1b351863d-snapshots\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.873436 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4af15a03-3a1e-4bb1-9f0d-c70ae28c325f-serving-cert\") pod \"service-ca-operator-5bcbc94c78-km55m\" (UID: \"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:09.873436 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6cnf\" (UniqueName: \"kubernetes.io/projected/4af15a03-3a1e-4bb1-9f0d-c70ae28c325f-kube-api-access-z6cnf\") pod \"service-ca-operator-5bcbc94c78-km55m\" (UID: \"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:09.873436 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcch6\" (UniqueName: \"kubernetes.io/projected/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-kube-api-access-vcch6\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.873436 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt9m5\" (UniqueName: \"kubernetes.io/projected/96025fea-bf2d-43e3-9506-7ebea486627c-kube-api-access-bt9m5\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:09.873436 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.873436 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:09.873436 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.873436 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41df392a-72e8-4b96-b6dd-a8eaf50b322b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-j5bf6\" (UID: \"41df392a-72e8-4b96-b6dd-a8eaf50b322b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.873907 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:09.873907 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-default-certificate\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.873907 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6mrj\" (UniqueName: \"kubernetes.io/projected/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-kube-api-access-q6mrj\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:09.873907 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.873561 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:10.373539058 +0000 UTC m=+66.672514080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : configmap references non-existent config key: service-ca.crt Jul 22 11:37:09.873907 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.873611 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 11:37:09.873907 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d0b446-3750-4f25-b181-87b1b351863d-service-ca-bundle\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.873907 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.873819 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:10.373802813 +0000 UTC m=+66.672777756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : secret "router-metrics-certs-default" not found Jul 22 11:37:09.873907 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw97d\" (UniqueName: \"kubernetes.io/projected/e0d0b446-3750-4f25-b181-87b1b351863d-kube-api-access-fw97d\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.873907 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4af15a03-3a1e-4bb1-9f0d-c70ae28c325f-config\") pod \"service-ca-operator-5bcbc94c78-km55m\" (UID: \"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:09.873907 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjhd\" (UniqueName: \"kubernetes.io/projected/62aac9b7-2b61-47c3-b68f-0cde3dee4534-kube-api-access-wjjhd\") pod \"network-check-source-7bc5b6bf85-r64wt\" (UID: \"62aac9b7-2b61-47c3-b68f-0cde3dee4534\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt" Jul 22 11:37:09.874426 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.873932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d0b446-3750-4f25-b181-87b1b351863d-tmp\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.874426 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.874211 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d0b446-3750-4f25-b181-87b1b351863d-tmp\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.874426 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.874247 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41df392a-72e8-4b96-b6dd-a8eaf50b322b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-j5bf6\" (UID: \"41df392a-72e8-4b96-b6dd-a8eaf50b322b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.874426 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.874272 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e0d0b446-3750-4f25-b181-87b1b351863d-snapshots\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.874817 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.874796 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d0b446-3750-4f25-b181-87b1b351863d-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.875586 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.875557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41df392a-72e8-4b96-b6dd-a8eaf50b322b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-j5bf6\" (UID: \"41df392a-72e8-4b96-b6dd-a8eaf50b322b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.876186 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.876110 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 11:37:09.876186 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.876127 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-stats-auth\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.876356 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.876235 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d0b446-3750-4f25-b181-87b1b351863d-serving-cert\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.878259 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.878239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-default-certificate\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.884509 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.884486 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 11:37:09.884677 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.884657 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcch6\" (UniqueName: \"kubernetes.io/projected/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-kube-api-access-vcch6\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:09.884756 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.884675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhx42\" (UniqueName: \"kubernetes.io/projected/41df392a-72e8-4b96-b6dd-a8eaf50b322b-kube-api-access-nhx42\") pod \"kube-storage-version-migrator-operator-7658d9b749-j5bf6\" (UID: \"41df392a-72e8-4b96-b6dd-a8eaf50b322b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.885231 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.885204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw97d\" (UniqueName: \"kubernetes.io/projected/e0d0b446-3750-4f25-b181-87b1b351863d-kube-api-access-fw97d\") pod \"insights-operator-555d9d7885-kzsvj\" (UID: \"e0d0b446-3750-4f25-b181-87b1b351863d\") " pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:09.894306 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.894287 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjjhd\" (UniqueName: \"kubernetes.io/projected/62aac9b7-2b61-47c3-b68f-0cde3dee4534-kube-api-access-wjjhd\") pod \"network-check-source-7bc5b6bf85-r64wt\" (UID: \"62aac9b7-2b61-47c3-b68f-0cde3dee4534\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt" Jul 22 11:37:09.895655 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.895630 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z85n\" (UniqueName: \"kubernetes.io/projected/60af647a-0dc7-4414-8cac-a61393b4cb51-kube-api-access-5z85n\") pod \"network-check-target-j7pfq\" (UID: \"60af647a-0dc7-4414-8cac-a61393b4cb51\") " pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:37:09.942737 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.942708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws" Jul 22 11:37:09.958841 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.958819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:09.972651 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.972632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" Jul 22 11:37:09.975103 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:37:09.975221 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/96025fea-bf2d-43e3-9506-7ebea486627c-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:09.975221 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4af15a03-3a1e-4bb1-9f0d-c70ae28c325f-serving-cert\") pod \"service-ca-operator-5bcbc94c78-km55m\" (UID: \"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:09.975344 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6cnf\" (UniqueName: \"kubernetes.io/projected/4af15a03-3a1e-4bb1-9f0d-c70ae28c325f-kube-api-access-z6cnf\") pod \"service-ca-operator-5bcbc94c78-km55m\" (UID: \"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:09.975344 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt9m5\" (UniqueName: \"kubernetes.io/projected/96025fea-bf2d-43e3-9506-7ebea486627c-kube-api-access-bt9m5\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:09.975344 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:09.975344 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:37:09.975670 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:09.975670 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6mrj\" (UniqueName: \"kubernetes.io/projected/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-kube-api-access-q6mrj\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:09.975670 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:37:09.975670 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4af15a03-3a1e-4bb1-9f0d-c70ae28c325f-config\") pod \"service-ca-operator-5bcbc94c78-km55m\" (UID: \"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:09.975670 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:37:09.975670 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.975510 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:09.975670 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.975577 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls podName:96025fea-bf2d-43e3-9506-7ebea486627c nodeName:}" failed. No retries permitted until 2026-07-22 11:37:10.475558939 +0000 UTC m=+66.774533878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4nc8" (UID: "96025fea-bf2d-43e3-9506-7ebea486627c") : secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:09.975670 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.975594 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 11:37:09.975670 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.975626 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls podName:869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:41.975616325 +0000 UTC m=+98.274591274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls") pod "dns-default-cp489" (UID: "869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2") : secret "dns-default-metrics-tls" not found Jul 22 11:37:09.976066 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.975688 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 11:37:09.976066 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.975698 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f98d8b84f-sgdh6: secret "image-registry-tls" not found Jul 22 11:37:09.976066 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.975730 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls podName:8ba37fbb-6fae-47d6-b308-b232339a177a nodeName:}" failed. No retries permitted until 2026-07-22 11:37:41.975719515 +0000 UTC m=+98.274694467 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls") pod "image-registry-f98d8b84f-sgdh6" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a") : secret "image-registry-tls" not found Jul 22 11:37:09.976066 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.975767 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:37:09.976066 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.975800 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls podName:e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa nodeName:}" failed. No retries permitted until 2026-07-22 11:37:10.475789354 +0000 UTC m=+66.774764292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bqc7c" (UID: "e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa") : secret "samples-operator-tls" not found Jul 22 11:37:09.976066 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.975981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/96025fea-bf2d-43e3-9506-7ebea486627c-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:09.976345 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.976175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4af15a03-3a1e-4bb1-9f0d-c70ae28c325f-config\") pod \"service-ca-operator-5bcbc94c78-km55m\" (UID: \"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:09.977287 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.977266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4af15a03-3a1e-4bb1-9f0d-c70ae28c325f-serving-cert\") pod \"service-ca-operator-5bcbc94c78-km55m\" (UID: \"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:09.977937 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.977919 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 11:37:09.978039 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.977986 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 11:37:09.984597 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.984572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6mrj\" (UniqueName: \"kubernetes.io/projected/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-kube-api-access-q6mrj\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:09.984685 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.984572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt9m5\" (UniqueName: \"kubernetes.io/projected/96025fea-bf2d-43e3-9506-7ebea486627c-kube-api-access-bt9m5\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:09.985570 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.985549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6cnf\" (UniqueName: \"kubernetes.io/projected/4af15a03-3a1e-4bb1-9f0d-c70ae28c325f-kube-api-access-z6cnf\") pod \"service-ca-operator-5bcbc94c78-km55m\" (UID: \"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:09.985630 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.985617 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 11:37:09.985695 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:09.985684 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs podName:64781c58-80c2-4b43-bc93-8bd1a7df80ca nodeName:}" failed. No retries permitted until 2026-07-22 11:38:13.985666738 +0000 UTC m=+130.284641677 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs") pod "network-metrics-daemon-x92mg" (UID: "64781c58-80c2-4b43-bc93-8bd1a7df80ca") : secret "metrics-daemon-secret" not found Jul 22 11:37:09.987933 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:09.987916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aca7ae8-e7fc-417d-840b-a7fa1155723f-original-pull-secret\") pod \"global-pull-secret-syncer-gcrmm\" (UID: \"1aca7ae8-e7fc-417d-840b-a7fa1155723f\") " pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:37:10.022228 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.021286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt" Jul 22 11:37:10.025116 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.024886 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4v9m8\"" Jul 22 11:37:10.032505 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.032423 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:37:10.053797 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.053725 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-kzsvj" Jul 22 11:37:10.112101 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.112000 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gcrmm" Jul 22 11:37:10.125873 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.125750 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws"] Jul 22 11:37:10.140611 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:10.140575 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a58d7a4_7171_4103_8e5f_1d921c7572af.slice/crio-e114b491b001fdad0a77d26849be96d17b508b5376b022fd8b115c86ccbe07a1 WatchSource:0}: Error finding container e114b491b001fdad0a77d26849be96d17b508b5376b022fd8b115c86ccbe07a1: Status 404 returned error can't find the container with id e114b491b001fdad0a77d26849be96d17b508b5376b022fd8b115c86ccbe07a1 Jul 22 11:37:10.145344 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.145247 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-t5smg"] Jul 22 11:37:10.158408 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.158145 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" Jul 22 11:37:10.188083 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.187436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6"] Jul 22 11:37:10.257424 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.257383 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt"] Jul 22 11:37:10.271565 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.271534 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j7pfq"] Jul 22 11:37:10.274591 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:10.274542 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60af647a_0dc7_4414_8cac_a61393b4cb51.slice/crio-7ce2700447cf111aed1dfdb51a93fb8785e3be46b67a9bea4e4182ac637f58b0 WatchSource:0}: Error finding container 7ce2700447cf111aed1dfdb51a93fb8785e3be46b67a9bea4e4182ac637f58b0: Status 404 returned error can't find the container with id 7ce2700447cf111aed1dfdb51a93fb8785e3be46b67a9bea4e4182ac637f58b0 Jul 22 11:37:10.280942 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.280913 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kzsvj"] Jul 22 11:37:10.285483 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:10.285447 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d0b446_3750_4f25_b181_87b1b351863d.slice/crio-3376a48db1caba7a617d8f9c9ebb8ab059f47b56d1172e1c40c82ab9b11b3bae WatchSource:0}: Error finding container 3376a48db1caba7a617d8f9c9ebb8ab059f47b56d1172e1c40c82ab9b11b3bae: Status 404 returned error can't find the container with id 3376a48db1caba7a617d8f9c9ebb8ab059f47b56d1172e1c40c82ab9b11b3bae Jul 22 11:37:10.317315 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.317284 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gcrmm"] Jul 22 11:37:10.320162 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:10.320136 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aca7ae8_e7fc_417d_840b_a7fa1155723f.slice/crio-633a067a60746acab2203f3bd225a5b586286814c12140e30506768531aa22e8 WatchSource:0}: Error finding container 633a067a60746acab2203f3bd225a5b586286814c12140e30506768531aa22e8: Status 404 returned error can't find the container with id 633a067a60746acab2203f3bd225a5b586286814c12140e30506768531aa22e8 Jul 22 11:37:10.343454 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.343425 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m"] Jul 22 11:37:10.346417 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:10.346393 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af15a03_3a1e_4bb1_9f0d_c70ae28c325f.slice/crio-85c04398a194e834d19f6e4dd8357e3e99dbce26a24fdd32d0e269b41fa53508 WatchSource:0}: Error finding container 85c04398a194e834d19f6e4dd8357e3e99dbce26a24fdd32d0e269b41fa53508: Status 404 returned error can't find the container with id 85c04398a194e834d19f6e4dd8357e3e99dbce26a24fdd32d0e269b41fa53508 Jul 22 11:37:10.379104 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.379025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:10.379104 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.379071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:10.379284 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:10.379188 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 11:37:10.379284 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:10.379231 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:11.379210876 +0000 UTC m=+67.678185825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : configmap references non-existent config key: service-ca.crt Jul 22 11:37:10.379284 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:10.379261 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:11.379251773 +0000 UTC m=+67.678226709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : secret "router-metrics-certs-default" not found Jul 22 11:37:10.445825 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.445787 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" event={"ID":"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f","Type":"ContainerStarted","Data":"85c04398a194e834d19f6e4dd8357e3e99dbce26a24fdd32d0e269b41fa53508"} Jul 22 11:37:10.446807 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.446787 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j7pfq" event={"ID":"60af647a-0dc7-4414-8cac-a61393b4cb51","Type":"ContainerStarted","Data":"7ce2700447cf111aed1dfdb51a93fb8785e3be46b67a9bea4e4182ac637f58b0"} Jul 22 11:37:10.447896 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.447873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt" event={"ID":"62aac9b7-2b61-47c3-b68f-0cde3dee4534","Type":"ContainerStarted","Data":"1d7bdaa2217369a7cef30fb7462fc8ae469c55291e282af525e8783088eb8aad"} Jul 22 11:37:10.448869 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.448851 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gcrmm" event={"ID":"1aca7ae8-e7fc-417d-840b-a7fa1155723f","Type":"ContainerStarted","Data":"633a067a60746acab2203f3bd225a5b586286814c12140e30506768531aa22e8"} Jul 22 11:37:10.449789 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.449772 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kzsvj" event={"ID":"e0d0b446-3750-4f25-b181-87b1b351863d","Type":"ContainerStarted","Data":"3376a48db1caba7a617d8f9c9ebb8ab059f47b56d1172e1c40c82ab9b11b3bae"} Jul 22 11:37:10.450767 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.450750 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" event={"ID":"41df392a-72e8-4b96-b6dd-a8eaf50b322b","Type":"ContainerStarted","Data":"7e1fc40c92a2a0af4a82010a5dfd8c137c84a7855bf7a0372148b3f74ca16e53"} Jul 22 11:37:10.451762 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.451744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws" event={"ID":"7a58d7a4-7171-4103-8e5f-1d921c7572af","Type":"ContainerStarted","Data":"e114b491b001fdad0a77d26849be96d17b508b5376b022fd8b115c86ccbe07a1"} Jul 22 11:37:10.452723 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.452704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" event={"ID":"a4ca318a-3b21-42da-9525-6e907b40970d","Type":"ContainerStarted","Data":"ca0803bb5ea4b270bd4a720a27cbbc1b0b5808cbcc41625d1beffbd63682a5d4"} Jul 22 11:37:10.480259 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.480232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:10.480360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:10.480280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:10.480395 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:10.480381 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:37:10.480395 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:10.480384 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:10.480453 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:10.480429 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls podName:e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa nodeName:}" failed. No retries permitted until 2026-07-22 11:37:11.480417537 +0000 UTC m=+67.779392485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bqc7c" (UID: "e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa") : secret "samples-operator-tls" not found Jul 22 11:37:10.480453 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:10.480444 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls podName:96025fea-bf2d-43e3-9506-7ebea486627c nodeName:}" failed. No retries permitted until 2026-07-22 11:37:11.480436234 +0000 UTC m=+67.779411169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4nc8" (UID: "96025fea-bf2d-43e3-9506-7ebea486627c") : secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:11.388635 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:11.388596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:11.389066 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:11.388664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:11.389066 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:11.388816 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 11:37:11.389066 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:11.388874 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:13.388856113 +0000 UTC m=+69.687831069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : secret "router-metrics-certs-default" not found Jul 22 11:37:11.389319 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:11.389308 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:13.389291913 +0000 UTC m=+69.688266869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : configmap references non-existent config key: service-ca.crt Jul 22 11:37:11.490629 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:11.489850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:11.490629 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:11.489925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:11.490629 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:11.490124 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:37:11.490629 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:11.490178 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls podName:e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa nodeName:}" failed. No retries permitted until 2026-07-22 11:37:13.490161774 +0000 UTC m=+69.789136714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bqc7c" (UID: "e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa") : secret "samples-operator-tls" not found Jul 22 11:37:11.490629 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:11.490551 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:11.490629 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:11.490597 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls podName:96025fea-bf2d-43e3-9506-7ebea486627c nodeName:}" failed. No retries permitted until 2026-07-22 11:37:13.490582259 +0000 UTC m=+69.789557220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4nc8" (UID: "96025fea-bf2d-43e3-9506-7ebea486627c") : secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:13.410104 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:13.409902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:13.410104 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:13.409973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:13.413610 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:13.411386 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:17.41135608 +0000 UTC m=+73.710331030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : configmap references non-existent config key: service-ca.crt Jul 22 11:37:13.413610 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:13.411795 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 11:37:13.413610 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:13.411877 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:17.411857372 +0000 UTC m=+73.710832322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : secret "router-metrics-certs-default" not found Jul 22 11:37:13.471377 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:13.471333 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws" event={"ID":"7a58d7a4-7171-4103-8e5f-1d921c7572af","Type":"ContainerStarted","Data":"475dcf12c01d8bf05334e9fad66732a8a1d6e0f2fdfaaf47a363163c2670ffd8"} Jul 22 11:37:13.489336 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:13.489038 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-xtxws" podStartSLOduration=2.279826258 podStartE2EDuration="4.489019183s" totalImagesPullingTime="2.209192925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:13.487385819 +0000 UTC m=+69.786360781" watchObservedRunningTime="2026-07-22 11:37:13.489019183 +0000 UTC m=+69.787994140" Jul 22 11:37:13.511635 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:13.511606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:13.511777 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:13.511676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:13.512641 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:13.512250 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:13.512641 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:13.512296 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:37:13.512641 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:13.512326 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls podName:96025fea-bf2d-43e3-9506-7ebea486627c nodeName:}" failed. No retries permitted until 2026-07-22 11:37:17.512307599 +0000 UTC m=+73.811282554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4nc8" (UID: "96025fea-bf2d-43e3-9506-7ebea486627c") : secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:13.512641 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:13.512345 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls podName:e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa nodeName:}" failed. No retries permitted until 2026-07-22 11:37:17.512334954 +0000 UTC m=+73.811309890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bqc7c" (UID: "e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa") : secret "samples-operator-tls" not found Jul 22 11:37:17.445609 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:17.445573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:17.446069 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:17.445645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:17.446069 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:17.445843 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 11:37:17.446069 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:17.445906 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:25.445887186 +0000 UTC m=+81.744862139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : secret "router-metrics-certs-default" not found Jul 22 11:37:17.446469 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:17.446337 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:25.446295615 +0000 UTC m=+81.745270552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : configmap references non-existent config key: service-ca.crt Jul 22 11:37:17.546888 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:17.546457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:17.546888 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:17.546547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:17.546888 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:17.546702 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:37:17.546888 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:17.546765 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls podName:e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa nodeName:}" failed. No retries permitted until 2026-07-22 11:37:25.546747671 +0000 UTC m=+81.845722629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bqc7c" (UID: "e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa") : secret "samples-operator-tls" not found Jul 22 11:37:17.546888 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:17.546829 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:17.546888 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:17.546864 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls podName:96025fea-bf2d-43e3-9506-7ebea486627c nodeName:}" failed. No retries permitted until 2026-07-22 11:37:25.546853275 +0000 UTC m=+81.845828223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4nc8" (UID: "96025fea-bf2d-43e3-9506-7ebea486627c") : secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:18.502842 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.502802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" event={"ID":"41df392a-72e8-4b96-b6dd-a8eaf50b322b","Type":"ContainerStarted","Data":"38bd5b98320f698339957b8f29cb2724ef3203bbec2b979011b95fd0f89f86e9"} Jul 22 11:37:18.504608 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.504581 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/0.log" Jul 22 11:37:18.504735 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.504627 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4ca318a-3b21-42da-9525-6e907b40970d" containerID="2445453aabdbba5f8eb10ea040d8256efa491e8b635501e39bea634884680338" exitCode=255 Jul 22 11:37:18.504801 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.504722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" event={"ID":"a4ca318a-3b21-42da-9525-6e907b40970d","Type":"ContainerDied","Data":"2445453aabdbba5f8eb10ea040d8256efa491e8b635501e39bea634884680338"} Jul 22 11:37:18.505140 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.505116 2579 scope.go:117] "RemoveContainer" containerID="2445453aabdbba5f8eb10ea040d8256efa491e8b635501e39bea634884680338" Jul 22 11:37:18.506301 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.506280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" event={"ID":"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f","Type":"ContainerStarted","Data":"d50f7a8e2f4ad50a4100a86d6b38c3e4f24ef46978022a2d71daa6021a24b8c9"} Jul 22 11:37:18.508072 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.508048 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j7pfq" event={"ID":"60af647a-0dc7-4414-8cac-a61393b4cb51","Type":"ContainerStarted","Data":"1385bb6a9d617451c101b86706103ba2964a4aab7363944a989c422a170bc1fa"} Jul 22 11:37:18.508172 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.508109 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:37:18.510302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.510280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt" event={"ID":"62aac9b7-2b61-47c3-b68f-0cde3dee4534","Type":"ContainerStarted","Data":"faf662e73c442091c4f476654713d2f659a867e55669ba3cdbdabc443cefb560"} Jul 22 11:37:18.512106 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.512058 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gcrmm" event={"ID":"1aca7ae8-e7fc-417d-840b-a7fa1155723f","Type":"ContainerStarted","Data":"f7f2ff79e4955702535017bc8f952b8c1a54b8dd08c5db57b18061c5769bdb44"} Jul 22 11:37:18.513997 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.513974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kzsvj" event={"ID":"e0d0b446-3750-4f25-b181-87b1b351863d","Type":"ContainerStarted","Data":"00310df120e7c9edf0d0dca5a7050eb23c4053bda1288000eb2bc90528579f73"} Jul 22 11:37:18.527040 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.526985 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" podStartSLOduration=2.376650463 podStartE2EDuration="9.526965212s" totalImagesPullingTime="7.150314749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:18.525578963 +0000 UTC m=+74.824553919" watchObservedRunningTime="2026-07-22 11:37:18.526965212 +0000 UTC m=+74.825940172" Jul 22 11:37:18.550268 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.549855 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gcrmm" podStartSLOduration=67.473207648 podStartE2EDuration="1m14.549835229s" totalImagesPullingTime="7.076627581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:18.54898976 +0000 UTC m=+74.847964718" watchObservedRunningTime="2026-07-22 11:37:18.549835229 +0000 UTC m=+74.848810186" Jul 22 11:37:18.567567 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.567509 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-r64wt" podStartSLOduration=2.464353954 podStartE2EDuration="9.567486419s" totalImagesPullingTime="7.103132465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:18.566368325 +0000 UTC m=+74.865343287" watchObservedRunningTime="2026-07-22 11:37:18.567486419 +0000 UTC m=+74.866461377" Jul 22 11:37:18.606774 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.605608 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-kzsvj" podStartSLOduration=2.516986615 podStartE2EDuration="9.605586444s" totalImagesPullingTime="7.088599829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:18.58799655 +0000 UTC m=+74.886971533" watchObservedRunningTime="2026-07-22 11:37:18.605586444 +0000 UTC m=+74.904561406" Jul 22 11:37:18.606774 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.606401 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j7pfq" podStartSLOduration=67.506882547 podStartE2EDuration="1m14.606386177s" totalImagesPullingTime="7.09950363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:18.604705465 +0000 UTC m=+74.903680462" watchObservedRunningTime="2026-07-22 11:37:18.606386177 +0000 UTC m=+74.905361137" Jul 22 11:37:18.645761 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:18.645605 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" podStartSLOduration=2.618200392 podStartE2EDuration="9.645583105s" totalImagesPullingTime="7.027382713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:18.645077044 +0000 UTC m=+74.944052002" watchObservedRunningTime="2026-07-22 11:37:18.645583105 +0000 UTC m=+74.944558064" Jul 22 11:37:19.520268 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.520242 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:37:19.520698 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.520679 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/0.log" Jul 22 11:37:19.520747 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.520723 2579 generic.go:356] "Generic (PLEG): container finished" podID="a4ca318a-3b21-42da-9525-6e907b40970d" containerID="d6cf0fff31c10a1995a4e68af0de2a9c53c66786bebad0a6814f1e56eaf63b06" exitCode=255 Jul 22 11:37:19.520848 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.520825 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" event={"ID":"a4ca318a-3b21-42da-9525-6e907b40970d","Type":"ContainerDied","Data":"d6cf0fff31c10a1995a4e68af0de2a9c53c66786bebad0a6814f1e56eaf63b06"} Jul 22 11:37:19.520892 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.520877 2579 scope.go:117] "RemoveContainer" containerID="2445453aabdbba5f8eb10ea040d8256efa491e8b635501e39bea634884680338" Jul 22 11:37:19.521115 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.521095 2579 scope.go:117] "RemoveContainer" containerID="d6cf0fff31c10a1995a4e68af0de2a9c53c66786bebad0a6814f1e56eaf63b06" Jul 22 11:37:19.521322 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:19.521296 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-t5smg_openshift-console-operator(a4ca318a-3b21-42da-9525-6e907b40970d)\"" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" podUID="a4ca318a-3b21-42da-9525-6e907b40970d" Jul 22 11:37:19.680568 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.680518 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v4wgf"] Jul 22 11:37:19.684617 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.684585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.687274 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.687250 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 11:37:19.687384 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.687281 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t7cck\"" Jul 22 11:37:19.687384 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.687305 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 11:37:19.693628 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.693603 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v4wgf"] Jul 22 11:37:19.768463 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.768438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.768463 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.768466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qqps\" (UniqueName: \"kubernetes.io/projected/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-api-access-9qqps\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.768680 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.768505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/386b36f3-f8d6-4928-a089-79e0b97db4f4-crio-socket\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.768680 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.768638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/386b36f3-f8d6-4928-a089-79e0b97db4f4-data-volume\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.768680 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.768660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.870117 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.870029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/386b36f3-f8d6-4928-a089-79e0b97db4f4-data-volume\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.870117 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.870068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.870117 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.870117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.870368 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.870142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qqps\" (UniqueName: \"kubernetes.io/projected/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-api-access-9qqps\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.870368 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:19.870274 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:37:19.870368 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:19.870349 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:37:19.870467 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:19.870351 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:20.370333902 +0000 UTC m=+76.669308838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : configmap "kube-rbac-proxy" not found Jul 22 11:37:19.870467 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:19.870418 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:20.370400533 +0000 UTC m=+76.669375470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : secret "insights-runtime-extractor-tls" not found Jul 22 11:37:19.870467 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.870442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/386b36f3-f8d6-4928-a089-79e0b97db4f4-crio-socket\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.870467 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.870456 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/386b36f3-f8d6-4928-a089-79e0b97db4f4-data-volume\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.870605 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.870512 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/386b36f3-f8d6-4928-a089-79e0b97db4f4-crio-socket\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.882475 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.882447 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qqps\" (UniqueName: \"kubernetes.io/projected/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-api-access-9qqps\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:19.959902 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.959860 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:19.960075 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:19.959928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:20.374538 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:20.374502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:20.374732 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:20.374558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:20.374732 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:20.374657 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:37:20.374732 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:20.374702 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:37:20.374732 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:20.374725 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:21.374709567 +0000 UTC m=+77.673684502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : secret "insights-runtime-extractor-tls" not found Jul 22 11:37:20.374863 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:20.374754 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:21.374741091 +0000 UTC m=+77.673716029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : configmap "kube-rbac-proxy" not found Jul 22 11:37:20.526965 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:20.526940 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:37:20.527352 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:20.527292 2579 scope.go:117] "RemoveContainer" containerID="d6cf0fff31c10a1995a4e68af0de2a9c53c66786bebad0a6814f1e56eaf63b06" Jul 22 11:37:20.527455 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:20.527435 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-t5smg_openshift-console-operator(a4ca318a-3b21-42da-9525-6e907b40970d)\"" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" podUID="a4ca318a-3b21-42da-9525-6e907b40970d" Jul 22 11:37:21.383540 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:21.383499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:21.383728 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:21.383559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:21.383728 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:21.383650 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:37:21.383728 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:21.383691 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:37:21.383728 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:21.383728 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:23.383702035 +0000 UTC m=+79.682676972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : secret "insights-runtime-extractor-tls" not found Jul 22 11:37:21.383866 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:21.383743 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:23.38373645 +0000 UTC m=+79.682711386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : configmap "kube-rbac-proxy" not found Jul 22 11:37:21.530542 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:21.530512 2579 scope.go:117] "RemoveContainer" containerID="d6cf0fff31c10a1995a4e68af0de2a9c53c66786bebad0a6814f1e56eaf63b06" Jul 22 11:37:21.530914 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:21.530666 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-t5smg_openshift-console-operator(a4ca318a-3b21-42da-9525-6e907b40970d)\"" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" podUID="a4ca318a-3b21-42da-9525-6e907b40970d" Jul 22 11:37:22.475747 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:22.475716 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ndr58_3d93eebd-7f1a-48fd-a0e3-51fe4ab46250/dns-node-resolver/0.log" Jul 22 11:37:23.401273 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:23.401238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:23.401662 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:23.401292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:23.401662 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:23.401367 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:37:23.401662 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:23.401389 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:37:23.401662 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:23.401431 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:27.401416937 +0000 UTC m=+83.700391874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : configmap "kube-rbac-proxy" not found Jul 22 11:37:23.401662 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:23.401451 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:27.401438892 +0000 UTC m=+83.700413829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : secret "insights-runtime-extractor-tls" not found Jul 22 11:37:23.675111 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:23.675032 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mwg4w_11037cad-c16a-4b04-b5ee-1287f5caef55/node-ca/0.log" Jul 22 11:37:24.274759 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:24.274726 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r45q9_7879951c-c75e-42ed-a411-5cebada8a304/serve-healthcheck-canary/0.log" Jul 22 11:37:25.076842 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:25.076807 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-j5bf6_41df392a-72e8-4b96-b6dd-a8eaf50b322b/kube-storage-version-migrator-operator/0.log" Jul 22 11:37:25.519572 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:25.519536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:25.519786 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:25.519586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:25.519786 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:25.519683 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 11:37:25.519786 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:25.519736 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:41.51971325 +0000 UTC m=+97.818688200 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : configmap references non-existent config key: service-ca.crt Jul 22 11:37:25.519786 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:25.519768 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs podName:1c5fe7bc-16c8-44d7-b77a-8929bc72ca26 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:41.519758053 +0000 UTC m=+97.818733004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs") pod "router-default-67fc875744-bt6br" (UID: "1c5fe7bc-16c8-44d7-b77a-8929bc72ca26") : secret "router-metrics-certs-default" not found Jul 22 11:37:25.619929 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:25.619892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:25.620092 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:25.620049 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 11:37:25.620092 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:25.620077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:25.620162 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:25.620138 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls podName:e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa nodeName:}" failed. No retries permitted until 2026-07-22 11:37:41.620122069 +0000 UTC m=+97.919097005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bqc7c" (UID: "e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa") : secret "samples-operator-tls" not found Jul 22 11:37:25.620250 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:25.620231 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:25.620303 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:25.620291 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls podName:96025fea-bf2d-43e3-9506-7ebea486627c nodeName:}" failed. No retries permitted until 2026-07-22 11:37:41.620275961 +0000 UTC m=+97.919250915 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4nc8" (UID: "96025fea-bf2d-43e3-9506-7ebea486627c") : secret "cluster-monitoring-operator-tls" not found Jul 22 11:37:27.438100 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:27.437953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:27.438100 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:27.438022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:27.438631 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:27.438102 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 11:37:27.438631 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:27.438162 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:35.438144679 +0000 UTC m=+91.737119628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : configmap "kube-rbac-proxy" not found Jul 22 11:37:27.438631 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:27.438103 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:37:27.438631 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:27.438249 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:35.43823577 +0000 UTC m=+91.737210706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : secret "insights-runtime-extractor-tls" not found Jul 22 11:37:32.200882 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:32.200848 2579 scope.go:117] "RemoveContainer" containerID="d6cf0fff31c10a1995a4e68af0de2a9c53c66786bebad0a6814f1e56eaf63b06" Jul 22 11:37:32.570126 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:32.570046 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:37:32.570126 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:32.570104 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" event={"ID":"a4ca318a-3b21-42da-9525-6e907b40970d","Type":"ContainerStarted","Data":"d489675b9e797a2e23d5380932b972842cff7112de65c25d09bb08481acdada9"} Jul 22 11:37:32.570414 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:32.570396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:32.589600 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:32.589544 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" podStartSLOduration=16.399976588 podStartE2EDuration="23.589527708s" totalImagesPullingTime="7.18955112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:32.588708755 +0000 UTC m=+88.887683714" watchObservedRunningTime="2026-07-22 11:37:32.589527708 +0000 UTC m=+88.888502694" Jul 22 11:37:33.166111 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:33.166082 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-t5smg" Jul 22 11:37:35.511577 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:35.511520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:35.511977 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:35.511658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:35.511977 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:35.511764 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 11:37:35.511977 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:35.511818 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls podName:386b36f3-f8d6-4928-a089-79e0b97db4f4 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:51.511802371 +0000 UTC m=+107.810777308 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4wgf" (UID: "386b36f3-f8d6-4928-a089-79e0b97db4f4") : secret "insights-runtime-extractor-tls" not found Jul 22 11:37:35.512091 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:35.512034 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/386b36f3-f8d6-4928-a089-79e0b97db4f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:41.556832 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.556799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:41.556832 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.556849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:41.557593 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.557568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-service-ca-bundle\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:41.559161 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.559143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5fe7bc-16c8-44d7-b77a-8929bc72ca26-metrics-certs\") pod \"router-default-67fc875744-bt6br\" (UID: \"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26\") " pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:41.569227 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.569203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:41.657743 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.657712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:41.657898 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.657778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:41.660162 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.660125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/96025fea-bf2d-43e3-9506-7ebea486627c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4nc8\" (UID: \"96025fea-bf2d-43e3-9506-7ebea486627c\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:41.660299 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.660160 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bqc7c\" (UID: \"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:41.694941 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.694910 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67fc875744-bt6br"] Jul 22 11:37:41.698768 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:41.698741 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5fe7bc_16c8_44d7_b77a_8929bc72ca26.slice/crio-60f413f84ac8257211c6960d6ffc21fdc24d004453f2e01f26e71c9c599370a6 WatchSource:0}: Error finding container 60f413f84ac8257211c6960d6ffc21fdc24d004453f2e01f26e71c9c599370a6: Status 404 returned error can't find the container with id 60f413f84ac8257211c6960d6ffc21fdc24d004453f2e01f26e71c9c599370a6 Jul 22 11:37:41.894064 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.894035 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" Jul 22 11:37:41.926042 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:41.926014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" Jul 22 11:37:42.028058 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.028028 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8"] Jul 22 11:37:42.031305 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:42.031274 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96025fea_bf2d_43e3_9506_7ebea486627c.slice/crio-4157ede0a2462f3b74549d8620f9119009f1b9c6612258f1958342a924734ca9 WatchSource:0}: Error finding container 4157ede0a2462f3b74549d8620f9119009f1b9c6612258f1958342a924734ca9: Status 404 returned error can't find the container with id 4157ede0a2462f3b74549d8620f9119009f1b9c6612258f1958342a924734ca9 Jul 22 11:37:42.061914 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.061886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:37:42.062051 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.061961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:37:42.064050 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.064026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2-metrics-tls\") pod \"dns-default-cp489\" (UID: \"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2\") " pod="openshift-dns/dns-default-cp489" Jul 22 11:37:42.064153 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.064087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"image-registry-f98d8b84f-sgdh6\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:37:42.072319 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.072294 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c"] Jul 22 11:37:42.292319 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.292237 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4jlt2\"" Jul 22 11:37:42.299711 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.299691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:37:42.310342 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.310321 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dg84n\"" Jul 22 11:37:42.317703 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.317675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cp489" Jul 22 11:37:42.437612 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.437576 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f98d8b84f-sgdh6"] Jul 22 11:37:42.441665 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:42.441575 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba37fbb_6fae_47d6_b308_b232339a177a.slice/crio-4aeba9618f33a867e950cd20c41000a505d3fb19f3606cc093cee6c83056bbbc WatchSource:0}: Error finding container 4aeba9618f33a867e950cd20c41000a505d3fb19f3606cc093cee6c83056bbbc: Status 404 returned error can't find the container with id 4aeba9618f33a867e950cd20c41000a505d3fb19f3606cc093cee6c83056bbbc Jul 22 11:37:42.452709 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.452684 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cp489"] Jul 22 11:37:42.455568 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:42.455546 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869c1d9a_ea5f_4ab4_a3f9_9c9a862985c2.slice/crio-eedc51e4d0e84709a04c7eeebf43985684a28833eb5f269f1c3198a01f7111de WatchSource:0}: Error finding container eedc51e4d0e84709a04c7eeebf43985684a28833eb5f269f1c3198a01f7111de: Status 404 returned error can't find the container with id eedc51e4d0e84709a04c7eeebf43985684a28833eb5f269f1c3198a01f7111de Jul 22 11:37:42.615172 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.615073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67fc875744-bt6br" event={"ID":"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26","Type":"ContainerStarted","Data":"fa4c6981ab01c411481bb2d17c3d7e5876c51049a62757eec7e626fbc120651a"} Jul 22 11:37:42.615172 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.615139 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67fc875744-bt6br" event={"ID":"1c5fe7bc-16c8-44d7-b77a-8929bc72ca26","Type":"ContainerStarted","Data":"60f413f84ac8257211c6960d6ffc21fdc24d004453f2e01f26e71c9c599370a6"} Jul 22 11:37:42.616718 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.616670 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cp489" event={"ID":"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2","Type":"ContainerStarted","Data":"eedc51e4d0e84709a04c7eeebf43985684a28833eb5f269f1c3198a01f7111de"} Jul 22 11:37:42.618526 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.618497 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" event={"ID":"8ba37fbb-6fae-47d6-b308-b232339a177a","Type":"ContainerStarted","Data":"9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2"} Jul 22 11:37:42.618652 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.618560 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:37:42.618652 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.618584 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" event={"ID":"8ba37fbb-6fae-47d6-b308-b232339a177a","Type":"ContainerStarted","Data":"4aeba9618f33a867e950cd20c41000a505d3fb19f3606cc093cee6c83056bbbc"} Jul 22 11:37:42.619851 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.619820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" event={"ID":"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa","Type":"ContainerStarted","Data":"4b21cbf9ac6163f09494c5280a97e211ff4fed9002d246ce9429589cb379cacc"} Jul 22 11:37:42.621557 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.621522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" event={"ID":"96025fea-bf2d-43e3-9506-7ebea486627c","Type":"ContainerStarted","Data":"4157ede0a2462f3b74549d8620f9119009f1b9c6612258f1958342a924734ca9"} Jul 22 11:37:42.636972 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.636901 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-67fc875744-bt6br" podStartSLOduration=33.636887356 podStartE2EDuration="33.636887356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:42.636102644 +0000 UTC m=+98.935077624" watchObservedRunningTime="2026-07-22 11:37:42.636887356 +0000 UTC m=+98.935862351" Jul 22 11:37:42.657410 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:42.657360 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" podStartSLOduration=98.657342588 podStartE2EDuration="1m38.657342588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:42.65617307 +0000 UTC m=+98.955148050" watchObservedRunningTime="2026-07-22 11:37:42.657342588 +0000 UTC m=+98.956317918" Jul 22 11:37:43.570292 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:43.570251 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:43.573291 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:43.573264 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:43.626769 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:43.626733 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:43.628330 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:43.628297 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-67fc875744-bt6br" Jul 22 11:37:46.138007 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.137924 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc"] Jul 22 11:37:46.164771 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.164744 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc"] Jul 22 11:37:46.164900 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.164830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc" Jul 22 11:37:46.167262 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.167241 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 22 11:37:46.167378 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.167267 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nddjv\"" Jul 22 11:37:46.300871 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.300838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7fc81cfe-5353-4f61-a55c-e096a5b1d34f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-srtkc\" (UID: \"7fc81cfe-5353-4f61-a55c-e096a5b1d34f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc" Jul 22 11:37:46.401621 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.401552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7fc81cfe-5353-4f61-a55c-e096a5b1d34f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-srtkc\" (UID: \"7fc81cfe-5353-4f61-a55c-e096a5b1d34f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc" Jul 22 11:37:46.404018 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.403990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7fc81cfe-5353-4f61-a55c-e096a5b1d34f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-srtkc\" (UID: \"7fc81cfe-5353-4f61-a55c-e096a5b1d34f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc" Jul 22 11:37:46.474063 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.474032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc" Jul 22 11:37:46.613598 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.613560 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc"] Jul 22 11:37:46.616498 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:46.616472 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc81cfe_5353_4f61_a55c_e096a5b1d34f.slice/crio-6eaf52378f1d38019bb366d4fb27d3b423ddc890cf1234a7c0765da354de4f3d WatchSource:0}: Error finding container 6eaf52378f1d38019bb366d4fb27d3b423ddc890cf1234a7c0765da354de4f3d: Status 404 returned error can't find the container with id 6eaf52378f1d38019bb366d4fb27d3b423ddc890cf1234a7c0765da354de4f3d Jul 22 11:37:46.638383 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.638352 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cp489" event={"ID":"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2","Type":"ContainerStarted","Data":"7e2ee6cce32311f76e1c5214e2699cd0009ed2031e517fcccd977aa73e5bc985"} Jul 22 11:37:46.638518 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.638418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cp489" event={"ID":"869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2","Type":"ContainerStarted","Data":"60de856e14d9ae42fcc652fc9095ff44790ee584cd11281f6217cde84bcc6003"} Jul 22 11:37:46.638518 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.638485 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cp489" Jul 22 11:37:46.640050 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.640028 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" event={"ID":"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa","Type":"ContainerStarted","Data":"efe034a8236b98021b84d0282e6b1378adc567ba08e5b599d13c7c3ffdc3beca"} Jul 22 11:37:46.640162 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.640076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" event={"ID":"e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa","Type":"ContainerStarted","Data":"c3a802057dd4f8073ee59686cdddabcd07c796990ae85695ad59a5a40ad542b7"} Jul 22 11:37:46.641166 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.641141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc" event={"ID":"7fc81cfe-5353-4f61-a55c-e096a5b1d34f","Type":"ContainerStarted","Data":"6eaf52378f1d38019bb366d4fb27d3b423ddc890cf1234a7c0765da354de4f3d"} Jul 22 11:37:46.642577 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.642556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" event={"ID":"96025fea-bf2d-43e3-9506-7ebea486627c","Type":"ContainerStarted","Data":"64e9307dc73688cbbd59e64578f68b0ed692c6a5c2f35619ce63ceb788cff91a"} Jul 22 11:37:46.659926 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.659444 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cp489" podStartSLOduration=65.542724751 podStartE2EDuration="1m8.659426104s" totalImagesPullingTime="3.116701353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:46.658846278 +0000 UTC m=+102.957821252" watchObservedRunningTime="2026-07-22 11:37:46.659426104 +0000 UTC m=+102.958401063" Jul 22 11:37:46.678724 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.678685 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4nc8" podStartSLOduration=34.138864826 podStartE2EDuration="37.678671581s" totalImagesPullingTime="3.539806755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:46.67760774 +0000 UTC m=+102.976582696" watchObservedRunningTime="2026-07-22 11:37:46.678671581 +0000 UTC m=+102.977646538" Jul 22 11:37:46.697523 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:46.697471 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bqc7c" podStartSLOduration=34.228283558 podStartE2EDuration="37.697453869s" totalImagesPullingTime="3.469170311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:46.696376854 +0000 UTC m=+102.995351811" watchObservedRunningTime="2026-07-22 11:37:46.697453869 +0000 UTC m=+102.996428841" Jul 22 11:37:48.652215 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:48.652157 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc" event={"ID":"7fc81cfe-5353-4f61-a55c-e096a5b1d34f","Type":"ContainerStarted","Data":"f2126b8dc2aab55ca1bbc8e91c83e5d489158787ef99e6ba949f5b0571711f1f"} Jul 22 11:37:48.652605 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:48.652248 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc" Jul 22 11:37:48.656723 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:48.656699 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc" Jul 22 11:37:48.668318 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:48.668270 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-srtkc" podStartSLOduration=1.2511747469999999 podStartE2EDuration="2.668253539s" totalImagesPullingTime="1.417078792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:48.667859163 +0000 UTC m=+104.966834123" watchObservedRunningTime="2026-07-22 11:37:48.668253539 +0000 UTC m=+104.967228495" Jul 22 11:37:49.523541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:49.523508 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j7pfq" Jul 22 11:37:51.544919 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:51.544872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:51.547129 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:51.547104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/386b36f3-f8d6-4928-a089-79e0b97db4f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4wgf\" (UID: \"386b36f3-f8d6-4928-a089-79e0b97db4f4\") " pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:51.795306 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:51.795215 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v4wgf" Jul 22 11:37:51.926305 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:51.926270 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v4wgf"] Jul 22 11:37:51.929269 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:51.929244 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386b36f3_f8d6_4928_a089_79e0b97db4f4.slice/crio-0c333ca0f0dbd3eaa1161f99180db42157c94c13c7c35bb11ecd9c31d5943eb9 WatchSource:0}: Error finding container 0c333ca0f0dbd3eaa1161f99180db42157c94c13c7c35bb11ecd9c31d5943eb9: Status 404 returned error can't find the container with id 0c333ca0f0dbd3eaa1161f99180db42157c94c13c7c35bb11ecd9c31d5943eb9 Jul 22 11:37:52.655636 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.655590 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b6c9b5789-7xzdj"] Jul 22 11:37:52.658913 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.658336 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.662469 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.662137 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 11:37:52.663125 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.663053 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rsvb8\"" Jul 22 11:37:52.668284 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.663302 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 11:37:52.668284 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.664227 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 11:37:52.668284 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.664432 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 11:37:52.668284 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.664435 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 11:37:52.668284 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.664925 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 11:37:52.668284 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.665120 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 11:37:52.670871 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.668751 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 11:37:52.671518 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.671417 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6c9b5789-7xzdj"] Jul 22 11:37:52.671625 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.671565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4wgf" event={"ID":"386b36f3-f8d6-4928-a089-79e0b97db4f4","Type":"ContainerStarted","Data":"0c333ca0f0dbd3eaa1161f99180db42157c94c13c7c35bb11ecd9c31d5943eb9"} Jul 22 11:37:52.755939 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.755900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-oauth-config\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.756140 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.755979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr2gc\" (UniqueName: \"kubernetes.io/projected/97e5e687-cabb-46b6-9535-c00eabfdf069-kube-api-access-pr2gc\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.756140 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.756029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-service-ca\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.756140 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.756065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-trusted-ca-bundle\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.756140 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.756128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-console-config\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.756405 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.756156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-serving-cert\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.756405 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.756212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-oauth-serving-cert\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.857646 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.857595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr2gc\" (UniqueName: \"kubernetes.io/projected/97e5e687-cabb-46b6-9535-c00eabfdf069-kube-api-access-pr2gc\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.857646 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.857648 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-service-ca\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.857896 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.857689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-trusted-ca-bundle\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.857896 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.857733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-console-config\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.857896 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.857769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-serving-cert\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.857896 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.857795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-oauth-serving-cert\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.857896 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.857810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-oauth-config\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.858970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.858916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-oauth-serving-cert\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.858970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.858959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-console-config\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.859558 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.859537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-service-ca\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.860295 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.860270 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-trusted-ca-bundle\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.861019 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.860995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-oauth-config\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.861647 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.861626 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-serving-cert\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.868149 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.868124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr2gc\" (UniqueName: \"kubernetes.io/projected/97e5e687-cabb-46b6-9535-c00eabfdf069-kube-api-access-pr2gc\") pod \"console-6b6c9b5789-7xzdj\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:52.981038 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:52.980917 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:37:53.137346 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:53.137310 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6c9b5789-7xzdj"] Jul 22 11:37:53.140721 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:53.140686 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e5e687_cabb_46b6_9535_c00eabfdf069.slice/crio-f7437e457fecb4300f6b2db81e1e0564d7fbcf5592d07f0e6a7208c451d0bb1a WatchSource:0}: Error finding container f7437e457fecb4300f6b2db81e1e0564d7fbcf5592d07f0e6a7208c451d0bb1a: Status 404 returned error can't find the container with id f7437e457fecb4300f6b2db81e1e0564d7fbcf5592d07f0e6a7208c451d0bb1a Jul 22 11:37:53.679184 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:53.679101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6c9b5789-7xzdj" event={"ID":"97e5e687-cabb-46b6-9535-c00eabfdf069","Type":"ContainerStarted","Data":"f7437e457fecb4300f6b2db81e1e0564d7fbcf5592d07f0e6a7208c451d0bb1a"} Jul 22 11:37:54.670336 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.670257 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vh88r"] Jul 22 11:37:54.677547 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.677334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.679592 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.679513 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 11:37:54.680090 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.679684 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-p6ght"] Jul 22 11:37:54.680090 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.679772 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 11:37:54.681838 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.680470 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 11:37:54.681838 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.681431 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 11:37:54.681838 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.681640 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g8l9j\"" Jul 22 11:37:54.682690 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.682670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.684994 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.684974 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 22 11:37:54.685739 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.685341 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kr9dx\"" Jul 22 11:37:54.685739 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.685402 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 22 11:37:54.685739 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.685501 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 22 11:37:54.690659 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.690627 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g"] Jul 22 11:37:54.693394 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.693373 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.694588 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.694566 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-p6ght"] Jul 22 11:37:54.695887 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.695863 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 11:37:54.697019 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.696095 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 11:37:54.697019 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.696490 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ldvmk\"" Jul 22 11:37:54.706616 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.706522 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g"] Jul 22 11:37:54.775616 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.775578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xplp9\" (UniqueName: \"kubernetes.io/projected/8dd28007-79e6-42a6-8794-40abf5fa6a21-kube-api-access-xplp9\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.775797 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.775635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.775797 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.775689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dd28007-79e6-42a6-8794-40abf5fa6a21-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.775797 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.775713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a86e44d1-3d0d-4e6f-947c-17736beed5dc-root\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.775797 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.775745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dd28007-79e6-42a6-8794-40abf5fa6a21-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.776016 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.775810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a86e44d1-3d0d-4e6f-947c-17736beed5dc-metrics-client-ca\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.776016 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.775933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-tls\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.776016 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.775963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-wtmp\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.776016 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.775981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.776016 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.775998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.776280 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.776080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx44c\" (UniqueName: \"kubernetes.io/projected/a86e44d1-3d0d-4e6f-947c-17736beed5dc-kube-api-access-wx44c\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.776280 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.776117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.776280 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.776155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.776280 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.776207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4l4m\" (UniqueName: \"kubernetes.io/projected/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-api-access-l4l4m\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.776280 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.776238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dd28007-79e6-42a6-8794-40abf5fa6a21-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.776280 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.776264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8758964-6f84-4261-b4f4-19c44ea2e7c0-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.776584 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.776314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a86e44d1-3d0d-4e6f-947c-17736beed5dc-sys\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.776584 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.776336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a8758964-6f84-4261-b4f4-19c44ea2e7c0-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.776584 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.776368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-textfile\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.876869 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.876824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-tls\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877058 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.876943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-wtmp\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877058 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.876981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877058 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.877058 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx44c\" (UniqueName: \"kubernetes.io/projected/a86e44d1-3d0d-4e6f-947c-17736beed5dc-kube-api-access-wx44c\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877276 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.877276 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.877276 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4l4m\" (UniqueName: \"kubernetes.io/projected/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-api-access-l4l4m\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.877276 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dd28007-79e6-42a6-8794-40abf5fa6a21-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.877276 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8758964-6f84-4261-b4f4-19c44ea2e7c0-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.877276 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:54.877226 2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 22 11:37:54.877276 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a86e44d1-3d0d-4e6f-947c-17736beed5dc-sys\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877503 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:54.877313 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-tls podName:a8758964-6f84-4261-b4f4-19c44ea2e7c0 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:55.377290331 +0000 UTC m=+111.676265271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-p6ght" (UID: "a8758964-6f84-4261-b4f4-19c44ea2e7c0") : secret "kube-state-metrics-tls" not found Jul 22 11:37:54.877503 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877332 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a86e44d1-3d0d-4e6f-947c-17736beed5dc-sys\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877503 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a8758964-6f84-4261-b4f4-19c44ea2e7c0-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.877503 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-wtmp\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877503 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-textfile\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877503 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xplp9\" (UniqueName: \"kubernetes.io/projected/8dd28007-79e6-42a6-8794-40abf5fa6a21-kube-api-access-xplp9\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.877503 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877849 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dd28007-79e6-42a6-8794-40abf5fa6a21-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.877849 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a86e44d1-3d0d-4e6f-947c-17736beed5dc-root\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877849 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dd28007-79e6-42a6-8794-40abf5fa6a21-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.877849 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a86e44d1-3d0d-4e6f-947c-17736beed5dc-metrics-client-ca\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877849 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-textfile\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.877849 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.877692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a8758964-6f84-4261-b4f4-19c44ea2e7c0-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.878112 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.878084 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a86e44d1-3d0d-4e6f-947c-17736beed5dc-metrics-client-ca\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.878364 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.878276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.879024 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.878559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a86e44d1-3d0d-4e6f-947c-17736beed5dc-root\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.879024 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:54.878646 2579 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 22 11:37:54.879024 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:37:54.878693 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd28007-79e6-42a6-8794-40abf5fa6a21-openshift-state-metrics-tls podName:8dd28007-79e6-42a6-8794-40abf5fa6a21 nodeName:}" failed. No retries permitted until 2026-07-22 11:37:55.378677327 +0000 UTC m=+111.677652278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8dd28007-79e6-42a6-8794-40abf5fa6a21-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-m4s4g" (UID: "8dd28007-79e6-42a6-8794-40abf5fa6a21") : secret "openshift-state-metrics-tls" not found Jul 22 11:37:54.879024 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.878641 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.879876 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.879848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dd28007-79e6-42a6-8794-40abf5fa6a21-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.879965 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.879917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-tls\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.880097 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.880075 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8758964-6f84-4261-b4f4-19c44ea2e7c0-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.881352 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.881327 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a86e44d1-3d0d-4e6f-947c-17736beed5dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.881969 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.881944 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dd28007-79e6-42a6-8794-40abf5fa6a21-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.882185 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.882156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.888448 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.888381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xplp9\" (UniqueName: \"kubernetes.io/projected/8dd28007-79e6-42a6-8794-40abf5fa6a21-kube-api-access-xplp9\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:54.889159 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.889112 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4l4m\" (UniqueName: \"kubernetes.io/projected/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-api-access-l4l4m\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:54.890171 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.890142 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx44c\" (UniqueName: \"kubernetes.io/projected/a86e44d1-3d0d-4e6f-947c-17736beed5dc-kube-api-access-wx44c\") pod \"node-exporter-vh88r\" (UID: \"a86e44d1-3d0d-4e6f-947c-17736beed5dc\") " pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:54.994897 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:54.994824 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vh88r" Jul 22 11:37:55.380627 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:55.380521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dd28007-79e6-42a6-8794-40abf5fa6a21-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:55.380627 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:55.380589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:55.383670 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:55.383643 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dd28007-79e6-42a6-8794-40abf5fa6a21-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-m4s4g\" (UID: \"8dd28007-79e6-42a6-8794-40abf5fa6a21\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:55.383802 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:55.383647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8758964-6f84-4261-b4f4-19c44ea2e7c0-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-p6ght\" (UID: \"a8758964-6f84-4261-b4f4-19c44ea2e7c0\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:55.606981 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:55.606948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" Jul 22 11:37:55.613076 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:55.613044 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" Jul 22 11:37:56.650256 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.650213 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cp489" Jul 22 11:37:56.684993 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.684962 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5dc67b8c86-4rngp"] Jul 22 11:37:56.689363 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.689333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.690883 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.690856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-grpc-tls\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.691017 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.690923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.691017 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.690953 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.691138 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.691049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrpm\" (UniqueName: \"kubernetes.io/projected/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-kube-api-access-lrrpm\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.691138 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.691126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.691262 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.691176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.691262 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.691252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-tls\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.691360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.691278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-metrics-client-ca\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.691742 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.691719 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 11:37:56.698514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.693125 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 11:37:56.698514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.693647 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 11:37:56.698514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.694290 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 11:37:56.698514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.694444 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1aa0guo1n71e7\"" Jul 22 11:37:56.698514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.696492 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 11:37:56.699221 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.699092 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pk4sf\"" Jul 22 11:37:56.702129 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.702103 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dc67b8c86-4rngp"] Jul 22 11:37:56.792161 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.792119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.792381 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.792175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.792381 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.792249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrpm\" (UniqueName: \"kubernetes.io/projected/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-kube-api-access-lrrpm\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.792381 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.792289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.792381 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.792338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.792586 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.792381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-tls\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.792586 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.792405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-metrics-client-ca\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.792586 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.792451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-grpc-tls\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.793402 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.793351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-metrics-client-ca\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.795549 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.795421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.795681 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.795651 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.795942 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.795919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.796015 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.795936 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-tls\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.796083 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.796062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-grpc-tls\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.796178 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.796113 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:56.800249 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:56.800225 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrpm\" (UniqueName: \"kubernetes.io/projected/6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c-kube-api-access-lrrpm\") pod \"thanos-querier-5dc67b8c86-4rngp\" (UID: \"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c\") " pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:57.008893 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:57.008796 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:37:58.209504 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:58.209425 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86e44d1_3d0d_4e6f_947c_17736beed5dc.slice/crio-714fcac65f11c306e9743313ebf2324092c7c048d2c3640e4b8b51d50a48b488 WatchSource:0}: Error finding container 714fcac65f11c306e9743313ebf2324092c7c048d2c3640e4b8b51d50a48b488: Status 404 returned error can't find the container with id 714fcac65f11c306e9743313ebf2324092c7c048d2c3640e4b8b51d50a48b488 Jul 22 11:37:58.375438 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.375172 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g"] Jul 22 11:37:58.393433 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.393319 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-p6ght"] Jul 22 11:37:58.397455 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:58.397315 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8758964_6f84_4261_b4f4_19c44ea2e7c0.slice/crio-3cd237b6ddd771bd203500607a036c69e1dade53e20344e3232976049a1e99ad WatchSource:0}: Error finding container 3cd237b6ddd771bd203500607a036c69e1dade53e20344e3232976049a1e99ad: Status 404 returned error can't find the container with id 3cd237b6ddd771bd203500607a036c69e1dade53e20344e3232976049a1e99ad Jul 22 11:37:58.418809 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.418714 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dc67b8c86-4rngp"] Jul 22 11:37:58.421764 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:58.421733 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6f48e3_cad1_4e36_8c64_e2fe10cb9e9c.slice/crio-97072cb0e076c0ae4979b3923be49f693bd79e42e3ca94f815933bf49c1beb1c WatchSource:0}: Error finding container 97072cb0e076c0ae4979b3923be49f693bd79e42e3ca94f815933bf49c1beb1c: Status 404 returned error can't find the container with id 97072cb0e076c0ae4979b3923be49f693bd79e42e3ca94f815933bf49c1beb1c Jul 22 11:37:58.707673 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.707581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" event={"ID":"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c","Type":"ContainerStarted","Data":"97072cb0e076c0ae4979b3923be49f693bd79e42e3ca94f815933bf49c1beb1c"} Jul 22 11:37:58.708890 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.708853 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" event={"ID":"a8758964-6f84-4261-b4f4-19c44ea2e7c0","Type":"ContainerStarted","Data":"3cd237b6ddd771bd203500607a036c69e1dade53e20344e3232976049a1e99ad"} Jul 22 11:37:58.710675 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.710573 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4wgf" event={"ID":"386b36f3-f8d6-4928-a089-79e0b97db4f4","Type":"ContainerStarted","Data":"dc0fc7ff37784180bc32b56d8960c4801fd3d018a53f970d1897bfc253e7c652"} Jul 22 11:37:58.712684 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.712617 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" event={"ID":"8dd28007-79e6-42a6-8794-40abf5fa6a21","Type":"ContainerStarted","Data":"85459ed9735f5bebb31f3391e5cc8df66970e3f3c86d849bbd1dd363568282c4"} Jul 22 11:37:58.712684 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.712676 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" event={"ID":"8dd28007-79e6-42a6-8794-40abf5fa6a21","Type":"ContainerStarted","Data":"465dfc957c8a38c1b1d32a0bd8ce21e561255cc5a879b58fa9f315a226b2660a"} Jul 22 11:37:58.712907 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.712706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" event={"ID":"8dd28007-79e6-42a6-8794-40abf5fa6a21","Type":"ContainerStarted","Data":"c7b4bc46ba9b1619c3aee69dc64cb145274a7c2185d8fda2ae7b6120addd1767"} Jul 22 11:37:58.713983 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.713955 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh88r" event={"ID":"a86e44d1-3d0d-4e6f-947c-17736beed5dc","Type":"ContainerStarted","Data":"714fcac65f11c306e9743313ebf2324092c7c048d2c3640e4b8b51d50a48b488"} Jul 22 11:37:58.715534 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.715509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6c9b5789-7xzdj" event={"ID":"97e5e687-cabb-46b6-9535-c00eabfdf069","Type":"ContainerStarted","Data":"776085eef5dd9d28c9a07e16db42ebfde73d8be4dbbe0af0120c161e8bd93417"} Jul 22 11:37:58.735842 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:58.735779 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b6c9b5789-7xzdj" podStartSLOduration=1.676668383 podStartE2EDuration="6.735757521s" totalImagesPullingTime="5.059089138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:37:58.73538995 +0000 UTC m=+115.034364927" watchObservedRunningTime="2026-07-22 11:37:58.735757521 +0000 UTC m=+115.034732478" Jul 22 11:37:59.019714 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.019689 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-676878bd5-xlljw"] Jul 22 11:37:59.022226 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.022206 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.025732 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.025708 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 11:37:59.025824 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.025747 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 11:37:59.025824 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.025787 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8vfvb\"" Jul 22 11:37:59.025824 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.025801 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 11:37:59.025978 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.025723 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 11:37:59.025978 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.025709 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ppkq7jecb85p\"" Jul 22 11:37:59.033754 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.033728 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-676878bd5-xlljw"] Jul 22 11:37:59.109898 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.109838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/997ce5d8-3759-4546-a771-72ccce27cd29-secret-metrics-server-tls\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.110083 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.109901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997ce5d8-3759-4546-a771-72ccce27cd29-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.110083 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.109947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/997ce5d8-3759-4546-a771-72ccce27cd29-client-ca-bundle\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.110083 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.110022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrcrf\" (UniqueName: \"kubernetes.io/projected/997ce5d8-3759-4546-a771-72ccce27cd29-kube-api-access-wrcrf\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.110083 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.110059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/997ce5d8-3759-4546-a771-72ccce27cd29-secret-metrics-server-client-certs\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.110357 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.110104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/997ce5d8-3759-4546-a771-72ccce27cd29-metrics-server-audit-profiles\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.110357 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.110134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/997ce5d8-3759-4546-a771-72ccce27cd29-audit-log\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.211610 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.211575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/997ce5d8-3759-4546-a771-72ccce27cd29-client-ca-bundle\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.212047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.211654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrcrf\" (UniqueName: \"kubernetes.io/projected/997ce5d8-3759-4546-a771-72ccce27cd29-kube-api-access-wrcrf\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.212047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.211693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/997ce5d8-3759-4546-a771-72ccce27cd29-secret-metrics-server-client-certs\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.212047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.211735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/997ce5d8-3759-4546-a771-72ccce27cd29-metrics-server-audit-profiles\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.212047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.211770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/997ce5d8-3759-4546-a771-72ccce27cd29-audit-log\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.212047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.211844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/997ce5d8-3759-4546-a771-72ccce27cd29-secret-metrics-server-tls\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.212047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.211869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997ce5d8-3759-4546-a771-72ccce27cd29-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.213569 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.212626 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997ce5d8-3759-4546-a771-72ccce27cd29-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.213569 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.213179 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/997ce5d8-3759-4546-a771-72ccce27cd29-audit-log\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.213569 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.213525 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/997ce5d8-3759-4546-a771-72ccce27cd29-metrics-server-audit-profiles\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.216381 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.216349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/997ce5d8-3759-4546-a771-72ccce27cd29-secret-metrics-server-client-certs\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.216490 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.216444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/997ce5d8-3759-4546-a771-72ccce27cd29-client-ca-bundle\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.217459 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.217435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/997ce5d8-3759-4546-a771-72ccce27cd29-secret-metrics-server-tls\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.224150 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.224092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrcrf\" (UniqueName: \"kubernetes.io/projected/997ce5d8-3759-4546-a771-72ccce27cd29-kube-api-access-wrcrf\") pod \"metrics-server-676878bd5-xlljw\" (UID: \"997ce5d8-3759-4546-a771-72ccce27cd29\") " pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.352862 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.352775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:37:59.726596 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.726563 2579 generic.go:356] "Generic (PLEG): container finished" podID="a86e44d1-3d0d-4e6f-947c-17736beed5dc" containerID="fa61b10e18e223ed79feba549926748589a7ba8a0485c75872dae27888bb0553" exitCode=0 Jul 22 11:37:59.727885 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.727809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh88r" event={"ID":"a86e44d1-3d0d-4e6f-947c-17736beed5dc","Type":"ContainerDied","Data":"fa61b10e18e223ed79feba549926748589a7ba8a0485c75872dae27888bb0553"} Jul 22 11:37:59.777272 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.777245 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-676878bd5-xlljw"] Jul 22 11:37:59.779603 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:37:59.779564 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997ce5d8_3759_4546_a771_72ccce27cd29.slice/crio-51b0d66c603adef76bc9dac7816843be798085e0a7bf54bcaaad366419d3deb0 WatchSource:0}: Error finding container 51b0d66c603adef76bc9dac7816843be798085e0a7bf54bcaaad366419d3deb0: Status 404 returned error can't find the container with id 51b0d66c603adef76bc9dac7816843be798085e0a7bf54bcaaad366419d3deb0 Jul 22 11:37:59.869115 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:37:59.869019 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6c9b5789-7xzdj"] Jul 22 11:38:00.735294 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:00.735226 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-676878bd5-xlljw" event={"ID":"997ce5d8-3759-4546-a771-72ccce27cd29","Type":"ContainerStarted","Data":"51b0d66c603adef76bc9dac7816843be798085e0a7bf54bcaaad366419d3deb0"} Jul 22 11:38:00.737548 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:00.737515 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" event={"ID":"8dd28007-79e6-42a6-8794-40abf5fa6a21","Type":"ContainerStarted","Data":"3f1fa520bd033c3f5c8df04238967bdb7f4f7481d4552f4b765e7ced751de7b2"} Jul 22 11:38:00.740893 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:00.740688 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh88r" event={"ID":"a86e44d1-3d0d-4e6f-947c-17736beed5dc","Type":"ContainerStarted","Data":"6eaef220f6e7a0e3eee29a960068eecc465b6ca4ceea31bde94989aa8ad7cad0"} Jul 22 11:38:00.740893 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:00.740757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh88r" event={"ID":"a86e44d1-3d0d-4e6f-947c-17736beed5dc","Type":"ContainerStarted","Data":"be3848caa73c333da57b97ef74141d79ce512adf1e5f074369459cd3843579d8"} Jul 22 11:38:00.757249 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:00.757164 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-m4s4g" podStartSLOduration=5.610749084 podStartE2EDuration="6.757144948s" totalImagesPullingTime="1.146395864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:38:00.755536398 +0000 UTC m=+117.054511356" watchObservedRunningTime="2026-07-22 11:38:00.757144948 +0000 UTC m=+117.056119908" Jul 22 11:38:00.780122 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:00.779721 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vh88r" podStartSLOduration=5.890785652 podStartE2EDuration="6.779705035s" totalImagesPullingTime="781.669467ms" totalInitContainerRuntime="107.249916ms" isStatefulPod=true podCreationTimestamp="2026-07-22 11:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:38:00.778364415 +0000 UTC m=+117.077339375" watchObservedRunningTime="2026-07-22 11:38:00.779705035 +0000 UTC m=+117.078679993" Jul 22 11:38:02.981899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:02.981861 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:38:03.632537 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:03.632513 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:38:03.762261 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:03.761075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" event={"ID":"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c","Type":"ContainerStarted","Data":"2b64469408ececa71a2791a178cac6c4685396ab7c63ac7afe495fbef2d68520"} Jul 22 11:38:03.764047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:03.763101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" event={"ID":"a8758964-6f84-4261-b4f4-19c44ea2e7c0","Type":"ContainerStarted","Data":"54bb29b39580ca43eb3d35647df69eab87d992ddaf806a89cd6b796a0e46d230"} Jul 22 11:38:03.765520 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:03.765426 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4wgf" event={"ID":"386b36f3-f8d6-4928-a089-79e0b97db4f4","Type":"ContainerStarted","Data":"58d708fe749b24132373b5b67f9a02138b5f7068302e4683fd813597642ab63a"} Jul 22 11:38:03.769019 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:03.768911 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-676878bd5-xlljw" event={"ID":"997ce5d8-3759-4546-a771-72ccce27cd29","Type":"ContainerStarted","Data":"985db81ec6fe9ad1ad6621c4b606fd8e26e29d9980c5245d3ec492161481ccec"} Jul 22 11:38:03.794175 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:03.793406 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-676878bd5-xlljw" podStartSLOduration=1.007201684 podStartE2EDuration="4.79338343s" totalImagesPullingTime="3.786181746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:38:03.792243351 +0000 UTC m=+120.091218335" watchObservedRunningTime="2026-07-22 11:38:03.79338343 +0000 UTC m=+120.092358390" Jul 22 11:38:04.781513 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:04.781470 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" event={"ID":"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c","Type":"ContainerStarted","Data":"fc390bd09ca49f7f93087a2bf5109389b30f4432ea0addd100fa68eca1859b35"} Jul 22 11:38:04.781937 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:04.781585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" event={"ID":"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c","Type":"ContainerStarted","Data":"b6d093908402a8ccffac842ca7e556afd6d87aa89c38f7ae5bfdca14d737fb30"} Jul 22 11:38:04.783876 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:04.783837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" event={"ID":"a8758964-6f84-4261-b4f4-19c44ea2e7c0","Type":"ContainerStarted","Data":"22ce90c3e06c3f52d049a726b9760c33a133c6d8351d972719f360a3cd518975"} Jul 22 11:38:04.784008 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:04.783898 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" event={"ID":"a8758964-6f84-4261-b4f4-19c44ea2e7c0","Type":"ContainerStarted","Data":"b5802b367ac027796229f38f8e19d2ee038b8d7ccd462208a275dee19623ee42"} Jul 22 11:38:04.807620 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:04.807568 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-p6ght" podStartSLOduration=6.065573608 podStartE2EDuration="10.807547402s" totalImagesPullingTime="4.741973794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:38:04.804653468 +0000 UTC m=+121.103628425" watchObservedRunningTime="2026-07-22 11:38:04.807547402 +0000 UTC m=+121.106522362" Jul 22 11:38:05.793337 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:05.793297 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" event={"ID":"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c","Type":"ContainerStarted","Data":"0d4d4ff2992f498c996e46f9babad17c290edc8c0cb670962b7e0d39c8ec1693"} Jul 22 11:38:05.793821 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:05.793385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" event={"ID":"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c","Type":"ContainerStarted","Data":"59e242583c872c1ebd79150e85512a5a1e71146d0e940b677692d31259637e2c"} Jul 22 11:38:05.793821 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:05.793423 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" event={"ID":"6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c","Type":"ContainerStarted","Data":"29c173c67632d4a076e05c4b926e5e0332de639269782780f57e962cfbf94f19"} Jul 22 11:38:05.823917 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:05.823868 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" podStartSLOduration=3.79543162 podStartE2EDuration="9.823854254s" totalImagesPullingTime="6.028422634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:37:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:38:05.822520564 +0000 UTC m=+122.121495541" watchObservedRunningTime="2026-07-22 11:38:05.823854254 +0000 UTC m=+122.122829212" Jul 22 11:38:06.801781 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:06.801726 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:38:07.114657 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:07.114582 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f98d8b84f-sgdh6"] Jul 22 11:38:07.807925 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:07.807860 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4wgf" event={"ID":"386b36f3-f8d6-4928-a089-79e0b97db4f4","Type":"ContainerStarted","Data":"92bdd1ee9bd0253de5d02a893d2e0f16236c16e5d7c80888acdffa11d04c113c"} Jul 22 11:38:07.829011 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:07.828947 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v4wgf" podStartSLOduration=33.735966898 podStartE2EDuration="48.828929453s" totalImagesPullingTime="15.092962555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 11:37:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:38:07.827063646 +0000 UTC m=+124.126038628" watchObservedRunningTime="2026-07-22 11:38:07.828929453 +0000 UTC m=+124.127904467" Jul 22 11:38:12.813397 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:12.813369 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5dc67b8c86-4rngp" Jul 22 11:38:14.052998 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:14.052959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:38:14.055323 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:14.055296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64781c58-80c2-4b43-bc93-8bd1a7df80ca-metrics-certs\") pod \"network-metrics-daemon-x92mg\" (UID: \"64781c58-80c2-4b43-bc93-8bd1a7df80ca\") " pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:38:14.218189 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:14.218159 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dd5dh\"" Jul 22 11:38:14.225625 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:14.225603 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x92mg" Jul 22 11:38:14.354000 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:14.353976 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x92mg"] Jul 22 11:38:14.356554 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:38:14.356523 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64781c58_80c2_4b43_bc93_8bd1a7df80ca.slice/crio-d74ea1cfdc7d6bd88bf93df14217a31fa35f1606cb0f52e37e0ebae76a7427a8 WatchSource:0}: Error finding container d74ea1cfdc7d6bd88bf93df14217a31fa35f1606cb0f52e37e0ebae76a7427a8: Status 404 returned error can't find the container with id d74ea1cfdc7d6bd88bf93df14217a31fa35f1606cb0f52e37e0ebae76a7427a8 Jul 22 11:38:14.837473 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:14.837435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x92mg" event={"ID":"64781c58-80c2-4b43-bc93-8bd1a7df80ca","Type":"ContainerStarted","Data":"d74ea1cfdc7d6bd88bf93df14217a31fa35f1606cb0f52e37e0ebae76a7427a8"} Jul 22 11:38:15.843295 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:15.843252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x92mg" event={"ID":"64781c58-80c2-4b43-bc93-8bd1a7df80ca","Type":"ContainerStarted","Data":"13e60a3a31ae16b5a6903b4acce554d202d10b4c03d065025188c7519ea899f4"} Jul 22 11:38:15.843662 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:15.843313 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x92mg" event={"ID":"64781c58-80c2-4b43-bc93-8bd1a7df80ca","Type":"ContainerStarted","Data":"ec8280149701873a6c352cb7a18a2866faf2e06816a3b2c21f30e9598973abfc"} Jul 22 11:38:15.861119 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:15.861070 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x92mg" podStartSLOduration=130.918034075 podStartE2EDuration="2m11.861054685s" totalImagesPullingTime="943.02061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:38:15.859818865 +0000 UTC m=+132.158793823" watchObservedRunningTime="2026-07-22 11:38:15.861054685 +0000 UTC m=+132.160029642" Jul 22 11:38:19.353900 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:19.353860 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:38:19.354404 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:19.353942 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:38:25.767101 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:25.767052 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b6c9b5789-7xzdj" podUID="97e5e687-cabb-46b6-9535-c00eabfdf069" containerName="console" containerID="cri-o://776085eef5dd9d28c9a07e16db42ebfde73d8be4dbbe0af0120c161e8bd93417" gracePeriod=15 Jul 22 11:38:25.888209 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:25.888162 2579 generic.go:356] "Generic (PLEG): container finished" podID="97e5e687-cabb-46b6-9535-c00eabfdf069" containerID="776085eef5dd9d28c9a07e16db42ebfde73d8be4dbbe0af0120c161e8bd93417" exitCode=0 Jul 22 11:38:25.888324 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:25.888236 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6c9b5789-7xzdj" event={"ID":"97e5e687-cabb-46b6-9535-c00eabfdf069","Type":"ContainerDied","Data":"776085eef5dd9d28c9a07e16db42ebfde73d8be4dbbe0af0120c161e8bd93417"} Jul 22 11:38:26.019741 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.019688 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:38:26.155147 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155120 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pr2gc\" (UniqueName: \"kubernetes.io/projected/97e5e687-cabb-46b6-9535-c00eabfdf069-kube-api-access-pr2gc\") pod \"97e5e687-cabb-46b6-9535-c00eabfdf069\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " Jul 22 11:38:26.155277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155174 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-console-config\") pod \"97e5e687-cabb-46b6-9535-c00eabfdf069\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " Jul 22 11:38:26.155277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155228 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-serving-cert\") pod \"97e5e687-cabb-46b6-9535-c00eabfdf069\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " Jul 22 11:38:26.155277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155260 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-oauth-config\") pod \"97e5e687-cabb-46b6-9535-c00eabfdf069\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " Jul 22 11:38:26.155444 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155290 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-trusted-ca-bundle\") pod \"97e5e687-cabb-46b6-9535-c00eabfdf069\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " Jul 22 11:38:26.155677 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155498 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-service-ca\") pod \"97e5e687-cabb-46b6-9535-c00eabfdf069\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " Jul 22 11:38:26.155677 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155580 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-oauth-serving-cert\") pod \"97e5e687-cabb-46b6-9535-c00eabfdf069\" (UID: \"97e5e687-cabb-46b6-9535-c00eabfdf069\") " Jul 22 11:38:26.155677 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155627 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-console-config" (OuterVolumeSpecName: "console-config") pod "97e5e687-cabb-46b6-9535-c00eabfdf069" (UID: "97e5e687-cabb-46b6-9535-c00eabfdf069"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:38:26.155877 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "97e5e687-cabb-46b6-9535-c00eabfdf069" (UID: "97e5e687-cabb-46b6-9535-c00eabfdf069"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:38:26.155877 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155845 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-console-config\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:26.155877 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155861 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-trusted-ca-bundle\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:26.156018 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.155991 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-service-ca" (OuterVolumeSpecName: "service-ca") pod "97e5e687-cabb-46b6-9535-c00eabfdf069" (UID: "97e5e687-cabb-46b6-9535-c00eabfdf069"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:38:26.156075 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.156023 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97e5e687-cabb-46b6-9535-c00eabfdf069" (UID: "97e5e687-cabb-46b6-9535-c00eabfdf069"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:38:26.157429 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.157394 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97e5e687-cabb-46b6-9535-c00eabfdf069" (UID: "97e5e687-cabb-46b6-9535-c00eabfdf069"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:38:26.157532 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.157454 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97e5e687-cabb-46b6-9535-c00eabfdf069" (UID: "97e5e687-cabb-46b6-9535-c00eabfdf069"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:38:26.157532 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.157475 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e5e687-cabb-46b6-9535-c00eabfdf069-kube-api-access-pr2gc" (OuterVolumeSpecName: "kube-api-access-pr2gc") pod "97e5e687-cabb-46b6-9535-c00eabfdf069" (UID: "97e5e687-cabb-46b6-9535-c00eabfdf069"). InnerVolumeSpecName "kube-api-access-pr2gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:38:26.256156 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.256131 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-oauth-config\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:26.256156 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.256153 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-service-ca\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:26.256298 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.256162 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97e5e687-cabb-46b6-9535-c00eabfdf069-oauth-serving-cert\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:26.256298 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.256171 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr2gc\" (UniqueName: \"kubernetes.io/projected/97e5e687-cabb-46b6-9535-c00eabfdf069-kube-api-access-pr2gc\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:26.256298 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.256181 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97e5e687-cabb-46b6-9535-c00eabfdf069-console-serving-cert\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:26.894778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.894746 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6c9b5789-7xzdj" Jul 22 11:38:26.894778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.894752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6c9b5789-7xzdj" event={"ID":"97e5e687-cabb-46b6-9535-c00eabfdf069","Type":"ContainerDied","Data":"f7437e457fecb4300f6b2db81e1e0564d7fbcf5592d07f0e6a7208c451d0bb1a"} Jul 22 11:38:26.895353 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.894807 2579 scope.go:117] "RemoveContainer" containerID="776085eef5dd9d28c9a07e16db42ebfde73d8be4dbbe0af0120c161e8bd93417" Jul 22 11:38:26.912571 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.912548 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6c9b5789-7xzdj"] Jul 22 11:38:26.922220 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:26.919253 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b6c9b5789-7xzdj"] Jul 22 11:38:28.205912 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:28.205879 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97e5e687-cabb-46b6-9535-c00eabfdf069" path="/var/lib/kubelet/pods/97e5e687-cabb-46b6-9535-c00eabfdf069/volumes" Jul 22 11:38:32.141791 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:32.141739 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" podUID="8ba37fbb-6fae-47d6-b308-b232339a177a" containerName="registry" containerID="cri-o://9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2" gracePeriod=30 Jul 22 11:38:32.904405 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:32.904382 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:38:32.920897 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:32.920871 2579 generic.go:356] "Generic (PLEG): container finished" podID="8ba37fbb-6fae-47d6-b308-b232339a177a" containerID="9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2" exitCode=0 Jul 22 11:38:32.921004 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:32.920912 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" Jul 22 11:38:32.921004 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:32.920953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" event={"ID":"8ba37fbb-6fae-47d6-b308-b232339a177a","Type":"ContainerDied","Data":"9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2"} Jul 22 11:38:32.921075 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:32.921022 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f98d8b84f-sgdh6" event={"ID":"8ba37fbb-6fae-47d6-b308-b232339a177a","Type":"ContainerDied","Data":"4aeba9618f33a867e950cd20c41000a505d3fb19f3606cc093cee6c83056bbbc"} Jul 22 11:38:32.921075 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:32.921044 2579 scope.go:117] "RemoveContainer" containerID="9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2" Jul 22 11:38:32.928840 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:32.928789 2579 scope.go:117] "RemoveContainer" containerID="9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2" Jul 22 11:38:32.929242 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:38:32.929179 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2\": container with ID starting with 9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2 not found: ID does not exist" containerID="9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2" Jul 22 11:38:32.929326 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:32.929258 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2"} err="failed to get container status \"9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2\": rpc error: code = NotFound desc = could not find container \"9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2\": container with ID starting with 9700a73a08ec0f23f08974f588ebf0f59c2c3ddb99d0dcc283a3626e187ffdc2 not found: ID does not exist" Jul 22 11:38:33.011361 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011338 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d8lj\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-kube-api-access-4d8lj\") pod \"8ba37fbb-6fae-47d6-b308-b232339a177a\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " Jul 22 11:38:33.011472 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011373 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-installation-pull-secrets\") pod \"8ba37fbb-6fae-47d6-b308-b232339a177a\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " Jul 22 11:38:33.011472 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011397 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-trusted-ca\") pod \"8ba37fbb-6fae-47d6-b308-b232339a177a\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " Jul 22 11:38:33.011472 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011420 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-certificates\") pod \"8ba37fbb-6fae-47d6-b308-b232339a177a\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " Jul 22 11:38:33.011636 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011475 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-bound-sa-token\") pod \"8ba37fbb-6fae-47d6-b308-b232339a177a\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " Jul 22 11:38:33.011636 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011515 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ba37fbb-6fae-47d6-b308-b232339a177a-ca-trust-extracted\") pod \"8ba37fbb-6fae-47d6-b308-b232339a177a\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " Jul 22 11:38:33.011636 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011554 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-image-registry-private-configuration\") pod \"8ba37fbb-6fae-47d6-b308-b232339a177a\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " Jul 22 11:38:33.011636 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011599 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") pod \"8ba37fbb-6fae-47d6-b308-b232339a177a\" (UID: \"8ba37fbb-6fae-47d6-b308-b232339a177a\") " Jul 22 11:38:33.011847 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011789 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8ba37fbb-6fae-47d6-b308-b232339a177a" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:38:33.011901 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011856 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8ba37fbb-6fae-47d6-b308-b232339a177a" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:38:33.011901 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.011878 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-trusted-ca\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:33.014014 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.013985 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8ba37fbb-6fae-47d6-b308-b232339a177a" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:38:33.014095 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.014035 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-kube-api-access-4d8lj" (OuterVolumeSpecName: "kube-api-access-4d8lj") pod "8ba37fbb-6fae-47d6-b308-b232339a177a" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a"). InnerVolumeSpecName "kube-api-access-4d8lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:38:33.014095 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.014058 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8ba37fbb-6fae-47d6-b308-b232339a177a" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:38:33.014237 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.014215 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8ba37fbb-6fae-47d6-b308-b232339a177a" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:38:33.014292 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.014220 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8ba37fbb-6fae-47d6-b308-b232339a177a" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:38:33.021874 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.021846 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba37fbb-6fae-47d6-b308-b232339a177a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8ba37fbb-6fae-47d6-b308-b232339a177a" (UID: "8ba37fbb-6fae-47d6-b308-b232339a177a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:38:33.112496 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.112474 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:33.112496 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.112496 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d8lj\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-kube-api-access-4d8lj\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:33.112625 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.112507 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-installation-pull-secrets\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:33.112625 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.112516 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ba37fbb-6fae-47d6-b308-b232339a177a-registry-certificates\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:33.112625 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.112524 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ba37fbb-6fae-47d6-b308-b232339a177a-bound-sa-token\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:33.112625 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.112532 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ba37fbb-6fae-47d6-b308-b232339a177a-ca-trust-extracted\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:33.112625 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.112541 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ba37fbb-6fae-47d6-b308-b232339a177a-image-registry-private-configuration\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:38:33.246561 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.246532 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f98d8b84f-sgdh6"] Jul 22 11:38:33.254792 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:33.254767 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f98d8b84f-sgdh6"] Jul 22 11:38:34.205343 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:34.205315 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba37fbb-6fae-47d6-b308-b232339a177a" path="/var/lib/kubelet/pods/8ba37fbb-6fae-47d6-b308-b232339a177a/volumes" Jul 22 11:38:39.358339 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:39.358271 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:38:39.362027 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:39.362007 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-676878bd5-xlljw" Jul 22 11:38:39.951123 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:39.951092 2579 generic.go:356] "Generic (PLEG): container finished" podID="e0d0b446-3750-4f25-b181-87b1b351863d" containerID="00310df120e7c9edf0d0dca5a7050eb23c4053bda1288000eb2bc90528579f73" exitCode=0 Jul 22 11:38:39.951302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:39.951165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kzsvj" event={"ID":"e0d0b446-3750-4f25-b181-87b1b351863d","Type":"ContainerDied","Data":"00310df120e7c9edf0d0dca5a7050eb23c4053bda1288000eb2bc90528579f73"} Jul 22 11:38:39.951624 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:39.951605 2579 scope.go:117] "RemoveContainer" containerID="00310df120e7c9edf0d0dca5a7050eb23c4053bda1288000eb2bc90528579f73" Jul 22 11:38:40.957001 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:40.956965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kzsvj" event={"ID":"e0d0b446-3750-4f25-b181-87b1b351863d","Type":"ContainerStarted","Data":"b8bcf65ccc24cb3b9b5e2650fc9add3c6bc6c18eff2e2e06446b9a01e743d3aa"} Jul 22 11:38:42.966731 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:42.966665 2579 generic.go:356] "Generic (PLEG): container finished" podID="41df392a-72e8-4b96-b6dd-a8eaf50b322b" containerID="38bd5b98320f698339957b8f29cb2724ef3203bbec2b979011b95fd0f89f86e9" exitCode=0 Jul 22 11:38:42.967064 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:42.966736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" event={"ID":"41df392a-72e8-4b96-b6dd-a8eaf50b322b","Type":"ContainerDied","Data":"38bd5b98320f698339957b8f29cb2724ef3203bbec2b979011b95fd0f89f86e9"} Jul 22 11:38:42.967104 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:42.967088 2579 scope.go:117] "RemoveContainer" containerID="38bd5b98320f698339957b8f29cb2724ef3203bbec2b979011b95fd0f89f86e9" Jul 22 11:38:43.972393 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:43.972351 2579 generic.go:356] "Generic (PLEG): container finished" podID="4af15a03-3a1e-4bb1-9f0d-c70ae28c325f" containerID="d50f7a8e2f4ad50a4100a86d6b38c3e4f24ef46978022a2d71daa6021a24b8c9" exitCode=0 Jul 22 11:38:43.972818 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:43.972426 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" event={"ID":"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f","Type":"ContainerDied","Data":"d50f7a8e2f4ad50a4100a86d6b38c3e4f24ef46978022a2d71daa6021a24b8c9"} Jul 22 11:38:43.972818 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:43.972802 2579 scope.go:117] "RemoveContainer" containerID="d50f7a8e2f4ad50a4100a86d6b38c3e4f24ef46978022a2d71daa6021a24b8c9" Jul 22 11:38:43.974567 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:43.974542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-j5bf6" event={"ID":"41df392a-72e8-4b96-b6dd-a8eaf50b322b","Type":"ContainerStarted","Data":"11ce19baff861565bd0508256ee60df6e0da52cceded844d1c65c6d729556ea9"} Jul 22 11:38:44.980835 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:38:44.980798 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-km55m" event={"ID":"4af15a03-3a1e-4bb1-9f0d-c70ae28c325f","Type":"ContainerStarted","Data":"4e777b66cfaf2207e1e879a27ddf61fafe238f44129a6ffb6811bb1f2cc2b06c"} Jul 22 11:39:23.105001 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.104953 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bdc98c6f8-wc4mr"] Jul 22 11:39:23.105588 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.105314 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e5e687-cabb-46b6-9535-c00eabfdf069" containerName="console" Jul 22 11:39:23.105588 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.105327 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e5e687-cabb-46b6-9535-c00eabfdf069" containerName="console" Jul 22 11:39:23.105588 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.105338 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba37fbb-6fae-47d6-b308-b232339a177a" containerName="registry" Jul 22 11:39:23.105588 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.105343 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba37fbb-6fae-47d6-b308-b232339a177a" containerName="registry" Jul 22 11:39:23.105588 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.105404 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba37fbb-6fae-47d6-b308-b232339a177a" containerName="registry" Jul 22 11:39:23.105588 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.105415 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e5e687-cabb-46b6-9535-c00eabfdf069" containerName="console" Jul 22 11:39:23.110232 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.110188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.112874 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.112849 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 11:39:23.113000 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.112878 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 11:39:23.113000 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.112914 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rsvb8\"" Jul 22 11:39:23.113895 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.113877 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 11:39:23.114030 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.113906 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 11:39:23.114143 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.113906 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 11:39:23.114143 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.113918 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 11:39:23.114283 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.113976 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 11:39:23.119350 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.119319 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdc98c6f8-wc4mr"] Jul 22 11:39:23.119459 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.119434 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 11:39:23.203729 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.203694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-oauth-serving-cert\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.203729 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.203731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fvwp\" (UniqueName: \"kubernetes.io/projected/30e75ef2-eb89-49b9-8963-ce0d3f416988-kube-api-access-4fvwp\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.203945 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.203759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-serving-cert\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.203945 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.203808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-service-ca\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.203945 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.203861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-trusted-ca-bundle\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.203945 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.203882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-config\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.203945 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.203903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-oauth-config\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.304978 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.304939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-oauth-serving-cert\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.304978 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.304979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fvwp\" (UniqueName: \"kubernetes.io/projected/30e75ef2-eb89-49b9-8963-ce0d3f416988-kube-api-access-4fvwp\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.305229 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.305007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-serving-cert\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.305229 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.305023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-service-ca\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.305229 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.305039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-trusted-ca-bundle\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.305229 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.305060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-config\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.305229 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.305087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-oauth-config\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.305753 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.305730 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-oauth-serving-cert\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.305853 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.305789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-service-ca\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.305987 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.305965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-config\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.306040 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.306020 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-trusted-ca-bundle\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.307562 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.307545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-serving-cert\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.307643 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.307562 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-oauth-config\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.313310 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.313290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fvwp\" (UniqueName: \"kubernetes.io/projected/30e75ef2-eb89-49b9-8963-ce0d3f416988-kube-api-access-4fvwp\") pod \"console-6bdc98c6f8-wc4mr\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.421594 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.421560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:23.546030 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:23.546003 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdc98c6f8-wc4mr"] Jul 22 11:39:23.548555 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:39:23.548526 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e75ef2_eb89_49b9_8963_ce0d3f416988.slice/crio-b9c9a9f7357b97f93272ca496ba0a3b5db35c617ddf1e6f6909e634e44ab6f5a WatchSource:0}: Error finding container b9c9a9f7357b97f93272ca496ba0a3b5db35c617ddf1e6f6909e634e44ab6f5a: Status 404 returned error can't find the container with id b9c9a9f7357b97f93272ca496ba0a3b5db35c617ddf1e6f6909e634e44ab6f5a Jul 22 11:39:24.141826 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:24.141791 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdc98c6f8-wc4mr" event={"ID":"30e75ef2-eb89-49b9-8963-ce0d3f416988","Type":"ContainerStarted","Data":"21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451"} Jul 22 11:39:24.142251 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:24.141844 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdc98c6f8-wc4mr" event={"ID":"30e75ef2-eb89-49b9-8963-ce0d3f416988","Type":"ContainerStarted","Data":"b9c9a9f7357b97f93272ca496ba0a3b5db35c617ddf1e6f6909e634e44ab6f5a"} Jul 22 11:39:24.161514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:24.161456 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bdc98c6f8-wc4mr" podStartSLOduration=1.161440787 podStartE2EDuration="1.161440787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:39:24.159658945 +0000 UTC m=+200.458633931" watchObservedRunningTime="2026-07-22 11:39:24.161440787 +0000 UTC m=+200.460415742" Jul 22 11:39:29.524419 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.524387 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bdc98c6f8-wc4mr"] Jul 22 11:39:29.554387 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.554361 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-695df56d48-vbxtc"] Jul 22 11:39:29.556546 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.556522 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.570631 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.570611 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695df56d48-vbxtc"] Jul 22 11:39:29.654551 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.654518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv9bb\" (UniqueName: \"kubernetes.io/projected/72a5c146-f3bb-40da-8bb5-b03704e76863-kube-api-access-pv9bb\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.654685 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.654564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-serving-cert\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.654685 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.654626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-oauth-config\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.654685 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.654649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-trusted-ca-bundle\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.654802 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.654711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-oauth-serving-cert\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.654802 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.654740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-console-config\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.654802 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.654793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-service-ca\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.755564 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.755537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-oauth-config\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.755712 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.755570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-trusted-ca-bundle\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.755712 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.755599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-oauth-serving-cert\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.755712 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.755622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-console-config\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.755712 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.755660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-service-ca\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.755712 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.755687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv9bb\" (UniqueName: \"kubernetes.io/projected/72a5c146-f3bb-40da-8bb5-b03704e76863-kube-api-access-pv9bb\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.755945 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.755730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-serving-cert\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.756407 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.756384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-oauth-serving-cert\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.756515 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.756488 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-service-ca\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.756560 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.756531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-console-config\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.756560 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.756545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-trusted-ca-bundle\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.757978 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.757949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-oauth-config\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.758227 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.758181 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-serving-cert\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.764762 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.764741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv9bb\" (UniqueName: \"kubernetes.io/projected/72a5c146-f3bb-40da-8bb5-b03704e76863-kube-api-access-pv9bb\") pod \"console-695df56d48-vbxtc\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.865904 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.865844 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:29.989471 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:29.989450 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695df56d48-vbxtc"] Jul 22 11:39:29.992072 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:39:29.992046 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a5c146_f3bb_40da_8bb5_b03704e76863.slice/crio-07dd80a44031cd613202c33efb237befb39b4ce02617b9839b67f46f6c3d9ccf WatchSource:0}: Error finding container 07dd80a44031cd613202c33efb237befb39b4ce02617b9839b67f46f6c3d9ccf: Status 404 returned error can't find the container with id 07dd80a44031cd613202c33efb237befb39b4ce02617b9839b67f46f6c3d9ccf Jul 22 11:39:30.166815 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:30.166777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695df56d48-vbxtc" event={"ID":"72a5c146-f3bb-40da-8bb5-b03704e76863","Type":"ContainerStarted","Data":"1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9"} Jul 22 11:39:30.166979 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:30.166833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695df56d48-vbxtc" event={"ID":"72a5c146-f3bb-40da-8bb5-b03704e76863","Type":"ContainerStarted","Data":"07dd80a44031cd613202c33efb237befb39b4ce02617b9839b67f46f6c3d9ccf"} Jul 22 11:39:30.191419 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:30.191354 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-695df56d48-vbxtc" podStartSLOduration=1.191334104 podStartE2EDuration="1.191334104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:39:30.188373346 +0000 UTC m=+206.487348293" watchObservedRunningTime="2026-07-22 11:39:30.191334104 +0000 UTC m=+206.490309061" Jul 22 11:39:33.421993 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:33.421963 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:39.866603 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:39.866559 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:39.867071 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:39.866630 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:39.871401 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:39.871374 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:40.215448 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:40.215415 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:39:54.543859 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.543788 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bdc98c6f8-wc4mr" podUID="30e75ef2-eb89-49b9-8963-ce0d3f416988" containerName="console" containerID="cri-o://21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451" gracePeriod=15 Jul 22 11:39:54.802876 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.802823 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:54.947921 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.947895 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-trusted-ca-bundle\") pod \"30e75ef2-eb89-49b9-8963-ce0d3f416988\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " Jul 22 11:39:54.948053 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.947935 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-oauth-serving-cert\") pod \"30e75ef2-eb89-49b9-8963-ce0d3f416988\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " Jul 22 11:39:54.948053 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.947957 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fvwp\" (UniqueName: \"kubernetes.io/projected/30e75ef2-eb89-49b9-8963-ce0d3f416988-kube-api-access-4fvwp\") pod \"30e75ef2-eb89-49b9-8963-ce0d3f416988\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " Jul 22 11:39:54.948053 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.947990 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-config\") pod \"30e75ef2-eb89-49b9-8963-ce0d3f416988\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " Jul 22 11:39:54.948240 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.948077 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-oauth-config\") pod \"30e75ef2-eb89-49b9-8963-ce0d3f416988\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " Jul 22 11:39:54.948240 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.948132 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-serving-cert\") pod \"30e75ef2-eb89-49b9-8963-ce0d3f416988\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " Jul 22 11:39:54.948240 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.948164 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-service-ca\") pod \"30e75ef2-eb89-49b9-8963-ce0d3f416988\" (UID: \"30e75ef2-eb89-49b9-8963-ce0d3f416988\") " Jul 22 11:39:54.948383 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.948302 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-config" (OuterVolumeSpecName: "console-config") pod "30e75ef2-eb89-49b9-8963-ce0d3f416988" (UID: "30e75ef2-eb89-49b9-8963-ce0d3f416988"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:39:54.948383 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.948318 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "30e75ef2-eb89-49b9-8963-ce0d3f416988" (UID: "30e75ef2-eb89-49b9-8963-ce0d3f416988"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:39:54.948541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.948515 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-trusted-ca-bundle\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:39:54.948656 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.948540 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-config\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:39:54.948724 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.948594 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "30e75ef2-eb89-49b9-8963-ce0d3f416988" (UID: "30e75ef2-eb89-49b9-8963-ce0d3f416988"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:39:54.948778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.948734 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-service-ca" (OuterVolumeSpecName: "service-ca") pod "30e75ef2-eb89-49b9-8963-ce0d3f416988" (UID: "30e75ef2-eb89-49b9-8963-ce0d3f416988"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:39:54.950388 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.950359 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "30e75ef2-eb89-49b9-8963-ce0d3f416988" (UID: "30e75ef2-eb89-49b9-8963-ce0d3f416988"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:39:54.950546 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.950474 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "30e75ef2-eb89-49b9-8963-ce0d3f416988" (UID: "30e75ef2-eb89-49b9-8963-ce0d3f416988"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:39:54.950546 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:54.950525 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e75ef2-eb89-49b9-8963-ce0d3f416988-kube-api-access-4fvwp" (OuterVolumeSpecName: "kube-api-access-4fvwp") pod "30e75ef2-eb89-49b9-8963-ce0d3f416988" (UID: "30e75ef2-eb89-49b9-8963-ce0d3f416988"). InnerVolumeSpecName "kube-api-access-4fvwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:39:55.049230 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.049189 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-oauth-config\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:39:55.049230 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.049224 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30e75ef2-eb89-49b9-8963-ce0d3f416988-console-serving-cert\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:39:55.049230 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.049233 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-service-ca\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:39:55.049385 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.049242 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30e75ef2-eb89-49b9-8963-ce0d3f416988-oauth-serving-cert\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:39:55.049385 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.049251 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fvwp\" (UniqueName: \"kubernetes.io/projected/30e75ef2-eb89-49b9-8963-ce0d3f416988-kube-api-access-4fvwp\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:39:55.272824 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.272795 2579 generic.go:356] "Generic (PLEG): container finished" podID="30e75ef2-eb89-49b9-8963-ce0d3f416988" containerID="21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451" exitCode=0 Jul 22 11:39:55.272935 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.272843 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdc98c6f8-wc4mr" Jul 22 11:39:55.272935 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.272879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdc98c6f8-wc4mr" event={"ID":"30e75ef2-eb89-49b9-8963-ce0d3f416988","Type":"ContainerDied","Data":"21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451"} Jul 22 11:39:55.272935 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.272925 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdc98c6f8-wc4mr" event={"ID":"30e75ef2-eb89-49b9-8963-ce0d3f416988","Type":"ContainerDied","Data":"b9c9a9f7357b97f93272ca496ba0a3b5db35c617ddf1e6f6909e634e44ab6f5a"} Jul 22 11:39:55.273033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.272948 2579 scope.go:117] "RemoveContainer" containerID="21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451" Jul 22 11:39:55.282926 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.282911 2579 scope.go:117] "RemoveContainer" containerID="21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451" Jul 22 11:39:55.283186 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:39:55.283170 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451\": container with ID starting with 21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451 not found: ID does not exist" containerID="21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451" Jul 22 11:39:55.283270 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.283210 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451"} err="failed to get container status \"21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451\": rpc error: code = NotFound desc = could not find container \"21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451\": container with ID starting with 21582135ad4165e9594821c273bb250c5e54be16a72037202facea8f25acb451 not found: ID does not exist" Jul 22 11:39:55.295845 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.295825 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bdc98c6f8-wc4mr"] Jul 22 11:39:55.299335 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:55.299315 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bdc98c6f8-wc4mr"] Jul 22 11:39:56.206244 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:39:56.206213 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e75ef2-eb89-49b9-8963-ce0d3f416988" path="/var/lib/kubelet/pods/30e75ef2-eb89-49b9-8963-ce0d3f416988/volumes" Jul 22 11:41:04.102350 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:41:04.102311 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:41:04.106266 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:41:04.106245 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:41:04.112919 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:41:04.112898 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 11:42:05.084102 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.084065 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc"] Jul 22 11:42:05.084524 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.084386 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e75ef2-eb89-49b9-8963-ce0d3f416988" containerName="console" Jul 22 11:42:05.084524 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.084399 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e75ef2-eb89-49b9-8963-ce0d3f416988" containerName="console" Jul 22 11:42:05.084524 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.084489 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e75ef2-eb89-49b9-8963-ce0d3f416988" containerName="console" Jul 22 11:42:05.087167 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.087150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" Jul 22 11:42:05.090308 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.090287 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 22 11:42:05.091533 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.091515 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-668b7\"" Jul 22 11:42:05.091625 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.091520 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 22 11:42:05.119564 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.119537 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc"] Jul 22 11:42:05.223245 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.223190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptvw\" (UniqueName: \"kubernetes.io/projected/ebc4b953-2a31-4f79-8157-ba40eb91ba5f-kube-api-access-hptvw\") pod \"feast-operator-controller-manager-7c86c97887-tmjzc\" (UID: \"ebc4b953-2a31-4f79-8157-ba40eb91ba5f\") " pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" Jul 22 11:42:05.324023 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.323978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hptvw\" (UniqueName: \"kubernetes.io/projected/ebc4b953-2a31-4f79-8157-ba40eb91ba5f-kube-api-access-hptvw\") pod \"feast-operator-controller-manager-7c86c97887-tmjzc\" (UID: \"ebc4b953-2a31-4f79-8157-ba40eb91ba5f\") " pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" Jul 22 11:42:05.332623 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.332592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptvw\" (UniqueName: \"kubernetes.io/projected/ebc4b953-2a31-4f79-8157-ba40eb91ba5f-kube-api-access-hptvw\") pod \"feast-operator-controller-manager-7c86c97887-tmjzc\" (UID: \"ebc4b953-2a31-4f79-8157-ba40eb91ba5f\") " pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" Jul 22 11:42:05.396378 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.396351 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" Jul 22 11:42:05.532998 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.532969 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc"] Jul 22 11:42:05.537020 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:42:05.536992 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc4b953_2a31_4f79_8157_ba40eb91ba5f.slice/crio-83349ca3827301ee232e3852ada3aa6fe141fb9887118ed5eb87ab25afdcea11 WatchSource:0}: Error finding container 83349ca3827301ee232e3852ada3aa6fe141fb9887118ed5eb87ab25afdcea11: Status 404 returned error can't find the container with id 83349ca3827301ee232e3852ada3aa6fe141fb9887118ed5eb87ab25afdcea11 Jul 22 11:42:05.538774 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.538756 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:42:05.779531 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:05.779454 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" event={"ID":"ebc4b953-2a31-4f79-8157-ba40eb91ba5f","Type":"ContainerStarted","Data":"83349ca3827301ee232e3852ada3aa6fe141fb9887118ed5eb87ab25afdcea11"} Jul 22 11:42:08.794565 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:08.794529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" event={"ID":"ebc4b953-2a31-4f79-8157-ba40eb91ba5f","Type":"ContainerStarted","Data":"efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4"} Jul 22 11:42:08.794926 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:08.794646 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" Jul 22 11:42:08.813820 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:08.813775 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" podStartSLOduration=1.251067066 podStartE2EDuration="3.813761223s" totalImagesPullingTime="2.562694157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:42:08.812779009 +0000 UTC m=+365.111753968" watchObservedRunningTime="2026-07-22 11:42:08.813761223 +0000 UTC m=+365.112736180" Jul 22 11:42:19.801966 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:19.801930 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" Jul 22 11:42:42.368049 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.368008 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s"] Jul 22 11:42:42.375290 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.375268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.377962 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.377940 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 22 11:42:42.378072 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.377975 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 22 11:42:42.378072 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.378061 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 22 11:42:42.378316 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.378301 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-khr59\"" Jul 22 11:42:42.378417 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.378325 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 22 11:42:42.378417 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.378365 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 22 11:42:42.378528 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.378414 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 22 11:42:42.378528 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.378413 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 22 11:42:42.383803 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.383784 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s"] Jul 22 11:42:42.489290 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.489262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-feast-data\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.489399 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.489299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-registry-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.489399 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.489363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.489399 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.489389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-offline-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.489556 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.489443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwl6\" (UniqueName: \"kubernetes.io/projected/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-kube-api-access-6qwl6\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.489556 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.489474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-online-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.590000 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.589980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.590108 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.590007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-offline-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.590108 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.590029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwl6\" (UniqueName: \"kubernetes.io/projected/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-kube-api-access-6qwl6\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.590108 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.590057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-online-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.590291 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:42:42.590115 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 22 11:42:42.590291 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.590124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-feast-data\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.590291 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.590155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-registry-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.590291 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:42:42.590209 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls podName:1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0 nodeName:}" failed. No retries permitted until 2026-07-22 11:42:43.090177559 +0000 UTC m=+399.389152495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls") pod "feast-simple-feast-setup-c874b989b-k2c9s" (UID: "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0") : secret "feast-simple-feast-setup-ui-tls" not found Jul 22 11:42:42.590291 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:42:42.590248 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 22 11:42:42.590551 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:42:42.590321 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-online-tls podName:1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0 nodeName:}" failed. No retries permitted until 2026-07-22 11:42:43.090302683 +0000 UTC m=+399.389277633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-online-tls") pod "feast-simple-feast-setup-c874b989b-k2c9s" (UID: "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0") : secret "feast-simple-feast-setup-online-tls" not found Jul 22 11:42:42.590551 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.590505 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-feast-data\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.592604 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.592573 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-offline-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.592715 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.592582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-registry-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:42.598964 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:42.598946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwl6\" (UniqueName: \"kubernetes.io/projected/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-kube-api-access-6qwl6\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:43.094548 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:43.094521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:43.094676 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:43.094560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-online-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:43.094676 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:42:43.094645 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 22 11:42:43.094750 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:42:43.094707 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls podName:1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0 nodeName:}" failed. No retries permitted until 2026-07-22 11:42:44.094693587 +0000 UTC m=+400.393668523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls") pod "feast-simple-feast-setup-c874b989b-k2c9s" (UID: "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0") : secret "feast-simple-feast-setup-ui-tls" not found Jul 22 11:42:43.096820 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:43.096800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-online-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:44.102008 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:44.101965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:44.104449 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:44.104423 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls\") pod \"feast-simple-feast-setup-c874b989b-k2c9s\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:44.200274 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:44.200250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:42:44.335109 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:44.335088 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s"] Jul 22 11:42:44.336987 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:42:44.336961 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d34edda_bdb1_4cf6_82d7_ce4ad59a63e0.slice/crio-56707219e73a651415c656af65e5ce6639c536731ec0a8125c63eac8be599145 WatchSource:0}: Error finding container 56707219e73a651415c656af65e5ce6639c536731ec0a8125c63eac8be599145: Status 404 returned error can't find the container with id 56707219e73a651415c656af65e5ce6639c536731ec0a8125c63eac8be599145 Jul 22 11:42:44.945352 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:42:44.945308 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerStarted","Data":"56707219e73a651415c656af65e5ce6639c536731ec0a8125c63eac8be599145"} Jul 22 11:43:09.064017 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:09.063930 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerStarted","Data":"ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494"} Jul 22 11:43:14.088112 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:14.088080 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerID="ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494" exitCode=0 Jul 22 11:43:14.088534 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:14.088155 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerDied","Data":"ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494"} Jul 22 11:43:15.094154 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:15.094117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerStarted","Data":"1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8"} Jul 22 11:43:20.119774 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:20.119744 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerID="1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8" exitCode=0 Jul 22 11:43:20.120219 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:20.119797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerDied","Data":"1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8"} Jul 22 11:43:20.130717 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:20.130695 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 22 11:43:21.130640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:21.130587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerStarted","Data":"12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71"} Jul 22 11:43:21.131109 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:21.130660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerStarted","Data":"0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975"} Jul 22 11:43:21.131109 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:21.130682 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerStarted","Data":"c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75"} Jul 22 11:43:21.131109 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:21.130699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerStarted","Data":"98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d"} Jul 22 11:43:21.158483 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:21.158420 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podStartSLOduration=14.682667752 podStartE2EDuration="39.158396199s" totalImagesPullingTime="24.475728447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:43:21.157147816 +0000 UTC m=+437.456122775" watchObservedRunningTime="2026-07-22 11:43:21.158396199 +0000 UTC m=+437.457371157" Jul 22 11:43:23.200903 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:23.200859 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:23.201511 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:23.201491 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:23.201654 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:23.201642 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:23.201963 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:23.201946 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:23.203329 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:23.203278 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jul 22 11:43:23.203329 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:23.203304 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.28:8816: connect: connection refused" Jul 22 11:43:23.203495 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:23.203313 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.28:6571: connect: connection refused" Jul 22 11:43:23.203495 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:23.203365 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jul 22 11:43:24.209142 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:24.209106 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:24.209142 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:24.209147 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:24.209690 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:24.209167 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:24.209690 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:24.209185 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:26.201164 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:26.201121 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jul 22 11:43:26.208025 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:26.208000 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:26.208157 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:26.208145 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:26.208376 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:26.208360 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:27.167549 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:27.167520 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:27.167704 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:27.167674 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:27.170132 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:27.170116 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:29.201438 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:29.201409 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:43:30.180644 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:43:30.180619 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:44:31.732489 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:31.732448 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s"] Jul 22 11:44:31.733087 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:31.732744 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="registry" containerID="cri-o://98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d" gracePeriod=30 Jul 22 11:44:31.733087 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:31.732799 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="online" containerID="cri-o://c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75" gracePeriod=30 Jul 22 11:44:31.733087 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:31.732744 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="ui" containerID="cri-o://12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71" gracePeriod=30 Jul 22 11:44:31.733087 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:31.732781 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="offline" containerID="cri-o://0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975" gracePeriod=30 Jul 22 11:44:32.050773 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.050699 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n"] Jul 22 11:44:32.054526 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.054505 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.057167 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.057144 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-nv2ck\"" Jul 22 11:44:32.069982 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.069954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n"] Jul 22 11:44:32.152539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.152503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8kpj\" (UniqueName: \"kubernetes.io/projected/28070ec7-6e93-4350-915e-3760778eef55-kube-api-access-p8kpj\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.152741 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.152574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-registry-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.152741 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.152595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-online-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.152741 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.152620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/28070ec7-6e93-4350-915e-3760778eef55-feast-data\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.152741 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.152690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-ui-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.152903 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.152748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-offline-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.253376 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.253349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-registry-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.253532 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.253382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-online-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.253532 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.253455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/28070ec7-6e93-4350-915e-3760778eef55-feast-data\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.254466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.253818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-ui-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.254466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.253862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-offline-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.254466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.253973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8kpj\" (UniqueName: \"kubernetes.io/projected/28070ec7-6e93-4350-915e-3760778eef55-kube-api-access-p8kpj\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.254466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.253989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/28070ec7-6e93-4350-915e-3760778eef55-feast-data\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.254466 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:44:32.254083 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 22 11:44:32.254466 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:44:32.254161 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 22 11:44:32.254466 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:44:32.254224 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-ui-tls podName:28070ec7-6e93-4350-915e-3760778eef55 nodeName:}" failed. No retries permitted until 2026-07-22 11:44:32.754171979 +0000 UTC m=+509.053146928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-ui-tls") pod "feast-simple-feast-setup-c874b989b-dkc2n" (UID: "28070ec7-6e93-4350-915e-3760778eef55") : secret "feast-simple-feast-setup-ui-tls" not found Jul 22 11:44:32.254466 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:44:32.254246 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-offline-tls podName:28070ec7-6e93-4350-915e-3760778eef55 nodeName:}" failed. No retries permitted until 2026-07-22 11:44:32.754234507 +0000 UTC m=+509.053209457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-offline-tls") pod "feast-simple-feast-setup-c874b989b-dkc2n" (UID: "28070ec7-6e93-4350-915e-3760778eef55") : secret "feast-simple-feast-setup-offline-tls" not found Jul 22 11:44:32.256014 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.255995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-registry-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.256621 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.256598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-online-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.263523 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.263503 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8kpj\" (UniqueName: \"kubernetes.io/projected/28070ec7-6e93-4350-915e-3760778eef55-kube-api-access-p8kpj\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.450187 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.450159 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerID="12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71" exitCode=0 Jul 22 11:44:32.450187 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.450185 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerID="c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75" exitCode=0 Jul 22 11:44:32.450353 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.450222 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerDied","Data":"12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71"} Jul 22 11:44:32.450353 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.450261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerDied","Data":"c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75"} Jul 22 11:44:32.758350 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.758310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-ui-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.758350 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.758350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-offline-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.760745 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.760721 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-offline-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.760745 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.760737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-ui-tls\") pod \"feast-simple-feast-setup-c874b989b-dkc2n\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:32.965378 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:32.965336 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:33.110822 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:33.110788 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n"] Jul 22 11:44:33.112557 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:44:33.112531 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28070ec7_6e93_4350_915e_3760778eef55.slice/crio-0d3441666f3cc86a23e180c6bac128d5738356cddf85356713843d71aba22508 WatchSource:0}: Error finding container 0d3441666f3cc86a23e180c6bac128d5738356cddf85356713843d71aba22508: Status 404 returned error can't find the container with id 0d3441666f3cc86a23e180c6bac128d5738356cddf85356713843d71aba22508 Jul 22 11:44:33.456265 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:33.456229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerStarted","Data":"4c155dffc3b8635356265ac695a3ead6dfe005cb4c6c3b9631d6e7e35219bd8c"} Jul 22 11:44:33.456442 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:33.456288 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerStarted","Data":"0d3441666f3cc86a23e180c6bac128d5738356cddf85356713843d71aba22508"} Jul 22 11:44:37.168016 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:37.167975 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jul 22 11:44:38.482816 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:38.482777 2579 generic.go:356] "Generic (PLEG): container finished" podID="28070ec7-6e93-4350-915e-3760778eef55" containerID="4c155dffc3b8635356265ac695a3ead6dfe005cb4c6c3b9631d6e7e35219bd8c" exitCode=0 Jul 22 11:44:38.483333 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:38.482855 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerDied","Data":"4c155dffc3b8635356265ac695a3ead6dfe005cb4c6c3b9631d6e7e35219bd8c"} Jul 22 11:44:39.490453 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:39.490417 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerStarted","Data":"7ea9fa750e52b0b8b750b2bed8c21ab059702e6d8ebe98499305e2e8e7c1ac2a"} Jul 22 11:44:40.180367 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:40.180320 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jul 22 11:44:44.514475 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:44.514443 2579 generic.go:356] "Generic (PLEG): container finished" podID="28070ec7-6e93-4350-915e-3760778eef55" containerID="7ea9fa750e52b0b8b750b2bed8c21ab059702e6d8ebe98499305e2e8e7c1ac2a" exitCode=0 Jul 22 11:44:44.514862 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:44.514521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerDied","Data":"7ea9fa750e52b0b8b750b2bed8c21ab059702e6d8ebe98499305e2e8e7c1ac2a"} Jul 22 11:44:45.525611 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:45.525567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerStarted","Data":"3fa0f2fe2edd59d8afcc013f0892ab9d04ba0249c65e00e8c1efea9af031f908"} Jul 22 11:44:45.526075 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:45.525629 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerStarted","Data":"9a01195e4ba0dd42ffbc75955b2db7479785c6c37cc426b38b21be0492d4001f"} Jul 22 11:44:45.526075 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:45.525650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerStarted","Data":"0ac3860f799a2fbf62c2af7cdf4a9d696fefdbd077981aaf032b152d38e64efd"} Jul 22 11:44:45.526075 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:45.525684 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerStarted","Data":"26594f7e16763a619c3212c262d06299f975b359870ef9b6b91ec5c039cd7e89"} Jul 22 11:44:45.555818 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:45.555753 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podStartSLOduration=13.555731291 podStartE2EDuration="13.555731291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:44:45.553781481 +0000 UTC m=+521.852756438" watchObservedRunningTime="2026-07-22 11:44:45.555731291 +0000 UTC m=+521.854706250" Jul 22 11:44:47.167890 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:47.167844 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jul 22 11:44:47.966008 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:47.965971 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:47.966334 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:47.966308 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:47.966468 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:47.966341 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:47.966468 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:47.966358 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:47.968359 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:47.968326 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.29:6571: connect: connection refused" Jul 22 11:44:47.968574 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:47.968349 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Jul 22 11:44:47.968653 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:47.968354 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Jul 22 11:44:47.968653 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:47.968408 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Jul 22 11:44:50.180795 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:50.180754 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jul 22 11:44:50.966913 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:50.966867 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:50.967215 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:50.967163 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:50.967602 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:50.967577 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:50.967786 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:50.967770 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:50.967834 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:50.967800 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:50.967875 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:50.967858 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:50.969367 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:50.969343 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:50.972610 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:50.972589 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:51.560454 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:51.560413 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:51.560454 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:51.560463 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:51.560991 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:51.560972 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:51.563502 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:51.563482 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:44:57.168030 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:57.167911 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jul 22 11:44:57.168502 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:44:57.168060 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:45:00.180789 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:00.180745 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jul 22 11:45:00.181242 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:00.180881 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:45:02.375680 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.375655 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:45:02.406832 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.406813 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-feast-data\") pod \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " Jul 22 11:45:02.406939 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.406851 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-offline-tls\") pod \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " Jul 22 11:45:02.406939 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.406875 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-registry-tls\") pod \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " Jul 22 11:45:02.406939 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.406912 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-online-tls\") pod \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " Jul 22 11:45:02.407083 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.406946 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qwl6\" (UniqueName: \"kubernetes.io/projected/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-kube-api-access-6qwl6\") pod \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " Jul 22 11:45:02.407083 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.406981 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls\") pod \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\" (UID: \"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0\") " Jul 22 11:45:02.407407 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.407360 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-feast-data" (OuterVolumeSpecName: "feast-data") pod "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" (UID: "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:45:02.409339 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.409261 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-online-tls" (OuterVolumeSpecName: "online-tls") pod "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" (UID: "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:45:02.409444 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.409334 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" (UID: "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:45:02.409444 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.409358 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" (UID: "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:45:02.409617 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.409602 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" (UID: "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:45:02.409737 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.409716 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-kube-api-access-6qwl6" (OuterVolumeSpecName: "kube-api-access-6qwl6") pod "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" (UID: "1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0"). InnerVolumeSpecName "kube-api-access-6qwl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:45:02.508514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.508468 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:45:02.508514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.508487 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:45:02.508514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.508496 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qwl6\" (UniqueName: \"kubernetes.io/projected/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-kube-api-access-6qwl6\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:45:02.508514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.508506 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-ui-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:45:02.508514 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.508514 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:45:02.508775 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.508523 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0-offline-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:45:02.619382 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.619358 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerID="0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975" exitCode=137 Jul 22 11:45:02.619382 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.619378 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerID="98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d" exitCode=137 Jul 22 11:45:02.619551 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.619426 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" Jul 22 11:45:02.619551 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.619432 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerDied","Data":"0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975"} Jul 22 11:45:02.619551 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.619479 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerDied","Data":"98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d"} Jul 22 11:45:02.619551 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.619493 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s" event={"ID":"1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0","Type":"ContainerDied","Data":"56707219e73a651415c656af65e5ce6639c536731ec0a8125c63eac8be599145"} Jul 22 11:45:02.619551 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.619511 2579 scope.go:117] "RemoveContainer" containerID="12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71" Jul 22 11:45:02.628412 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.628397 2579 scope.go:117] "RemoveContainer" containerID="0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975" Jul 22 11:45:02.635432 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.635415 2579 scope.go:117] "RemoveContainer" containerID="c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75" Jul 22 11:45:02.642862 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.642837 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s"] Jul 22 11:45:02.642914 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.642888 2579 scope.go:117] "RemoveContainer" containerID="98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d" Jul 22 11:45:02.649256 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.649235 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-k2c9s"] Jul 22 11:45:02.650989 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.650973 2579 scope.go:117] "RemoveContainer" containerID="1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8" Jul 22 11:45:02.657735 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.657721 2579 scope.go:117] "RemoveContainer" containerID="ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494" Jul 22 11:45:02.664981 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.664955 2579 scope.go:117] "RemoveContainer" containerID="12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71" Jul 22 11:45:02.665240 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:45:02.665220 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71\": container with ID starting with 12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71 not found: ID does not exist" containerID="12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71" Jul 22 11:45:02.665289 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.665247 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71"} err="failed to get container status \"12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71\": rpc error: code = NotFound desc = could not find container \"12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71\": container with ID starting with 12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71 not found: ID does not exist" Jul 22 11:45:02.665289 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.665262 2579 scope.go:117] "RemoveContainer" containerID="0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975" Jul 22 11:45:02.665508 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:45:02.665490 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975\": container with ID starting with 0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975 not found: ID does not exist" containerID="0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975" Jul 22 11:45:02.665559 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.665512 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975"} err="failed to get container status \"0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975\": rpc error: code = NotFound desc = could not find container \"0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975\": container with ID starting with 0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975 not found: ID does not exist" Jul 22 11:45:02.665559 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.665529 2579 scope.go:117] "RemoveContainer" containerID="c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75" Jul 22 11:45:02.665747 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:45:02.665731 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75\": container with ID starting with c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75 not found: ID does not exist" containerID="c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75" Jul 22 11:45:02.665784 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.665751 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75"} err="failed to get container status \"c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75\": rpc error: code = NotFound desc = could not find container \"c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75\": container with ID starting with c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75 not found: ID does not exist" Jul 22 11:45:02.665784 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.665764 2579 scope.go:117] "RemoveContainer" containerID="98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d" Jul 22 11:45:02.665962 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:45:02.665947 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d\": container with ID starting with 98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d not found: ID does not exist" containerID="98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d" Jul 22 11:45:02.666009 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.665964 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d"} err="failed to get container status \"98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d\": rpc error: code = NotFound desc = could not find container \"98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d\": container with ID starting with 98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d not found: ID does not exist" Jul 22 11:45:02.666009 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.665975 2579 scope.go:117] "RemoveContainer" containerID="1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8" Jul 22 11:45:02.666210 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:45:02.666179 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8\": container with ID starting with 1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8 not found: ID does not exist" containerID="1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8" Jul 22 11:45:02.666255 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.666217 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8"} err="failed to get container status \"1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8\": rpc error: code = NotFound desc = could not find container \"1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8\": container with ID starting with 1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8 not found: ID does not exist" Jul 22 11:45:02.666255 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.666231 2579 scope.go:117] "RemoveContainer" containerID="ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494" Jul 22 11:45:02.666457 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:45:02.666436 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494\": container with ID starting with ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494 not found: ID does not exist" containerID="ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494" Jul 22 11:45:02.666505 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.666462 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494"} err="failed to get container status \"ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494\": rpc error: code = NotFound desc = could not find container \"ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494\": container with ID starting with ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494 not found: ID does not exist" Jul 22 11:45:02.666505 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.666477 2579 scope.go:117] "RemoveContainer" containerID="12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71" Jul 22 11:45:02.666671 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.666654 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71"} err="failed to get container status \"12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71\": rpc error: code = NotFound desc = could not find container \"12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71\": container with ID starting with 12dad25e84bd621532fd6c63b1dbab4589902e54360ffca15041d89d71feaf71 not found: ID does not exist" Jul 22 11:45:02.666715 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.666671 2579 scope.go:117] "RemoveContainer" containerID="0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975" Jul 22 11:45:02.666867 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.666852 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975"} err="failed to get container status \"0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975\": rpc error: code = NotFound desc = could not find container \"0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975\": container with ID starting with 0e0201928276157d556ae77bedb5d4546f5688ed543131d8d1d7259741228975 not found: ID does not exist" Jul 22 11:45:02.666909 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.666867 2579 scope.go:117] "RemoveContainer" containerID="c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75" Jul 22 11:45:02.667038 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.667023 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75"} err="failed to get container status \"c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75\": rpc error: code = NotFound desc = could not find container \"c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75\": container with ID starting with c272fe2efb791925b47f72bb39ae06b64bd6f159887ee385a4045c78b2a4af75 not found: ID does not exist" Jul 22 11:45:02.667085 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.667038 2579 scope.go:117] "RemoveContainer" containerID="98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d" Jul 22 11:45:02.667268 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.667245 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d"} err="failed to get container status \"98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d\": rpc error: code = NotFound desc = could not find container \"98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d\": container with ID starting with 98c01053955cac3f9a245915986d83fa9211f59e3d0fbb5513e91e385a990d4d not found: ID does not exist" Jul 22 11:45:02.667323 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.667270 2579 scope.go:117] "RemoveContainer" containerID="1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8" Jul 22 11:45:02.667482 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.667466 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8"} err="failed to get container status \"1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8\": rpc error: code = NotFound desc = could not find container \"1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8\": container with ID starting with 1aa976dbb167d856ae7b0f172b6d0dca20ebaad991781919e7d87b25bef2bcd8 not found: ID does not exist" Jul 22 11:45:02.667524 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.667483 2579 scope.go:117] "RemoveContainer" containerID="ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494" Jul 22 11:45:02.667686 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:02.667670 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494"} err="failed to get container status \"ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494\": rpc error: code = NotFound desc = could not find container \"ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494\": container with ID starting with ad8c64561aacb0e07783eb5284c9a0aee5207963ea7909814384254296205494 not found: ID does not exist" Jul 22 11:45:04.207131 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:04.207103 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" path="/var/lib/kubelet/pods/1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0/volumes" Jul 22 11:45:53.784505 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784473 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9"] Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784782 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="registry" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784793 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="registry" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784801 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="online" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784807 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="online" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784814 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="feast-init" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784820 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="feast-init" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784832 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="feast-apply" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784837 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="feast-apply" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784843 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="offline" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784848 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="offline" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784864 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="ui" Jul 22 11:45:53.784899 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784868 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="ui" Jul 22 11:45:53.785266 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784919 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="offline" Jul 22 11:45:53.785266 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784929 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="ui" Jul 22 11:45:53.785266 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784934 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="registry" Jul 22 11:45:53.785266 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.784940 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d34edda-bdb1-4cf6-82d7-ce4ad59a63e0" containerName="online" Jul 22 11:45:53.788665 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.788646 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.791421 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.791398 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-slk6s\"" Jul 22 11:45:53.791555 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.791537 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 22 11:45:53.791628 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.791608 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 22 11:45:53.791915 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.791894 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 22 11:45:53.792409 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.792390 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 22 11:45:53.792491 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.792473 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 22 11:45:53.792549 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.792498 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 22 11:45:53.792549 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.792504 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 22 11:45:53.801060 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.801040 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9"] Jul 22 11:45:53.883706 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.883680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/839ba787-1aaf-4b21-9cab-007c2b53f42e-registry-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.883825 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.883718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-offline-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.883825 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.883744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wltw2\" (UniqueName: \"kubernetes.io/projected/839ba787-1aaf-4b21-9cab-007c2b53f42e-kube-api-access-wltw2\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.883940 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.883842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-online-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.883940 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.883877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-ui-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.883940 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.883910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/839ba787-1aaf-4b21-9cab-007c2b53f42e-feast-data\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.985054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.985032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/839ba787-1aaf-4b21-9cab-007c2b53f42e-feast-data\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.985163 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.985068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/839ba787-1aaf-4b21-9cab-007c2b53f42e-registry-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.985163 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.985090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-offline-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.985163 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.985110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltw2\" (UniqueName: \"kubernetes.io/projected/839ba787-1aaf-4b21-9cab-007c2b53f42e-kube-api-access-wltw2\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.985163 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.985150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-online-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.985367 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.985168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-ui-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.985367 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:45:53.985285 2579 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jul 22 11:45:53.985367 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:45:53.985345 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-online-tls podName:839ba787-1aaf-4b21-9cab-007c2b53f42e nodeName:}" failed. No retries permitted until 2026-07-22 11:45:54.485326633 +0000 UTC m=+590.784301574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-online-tls") pod "feast-simple-feast-remote-setup-6599699bfc-7fmv9" (UID: "839ba787-1aaf-4b21-9cab-007c2b53f42e") : secret "feast-simple-feast-remote-setup-online-tls" not found Jul 22 11:45:53.985516 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.985449 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/839ba787-1aaf-4b21-9cab-007c2b53f42e-feast-data\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.985817 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.985798 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/839ba787-1aaf-4b21-9cab-007c2b53f42e-registry-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.987586 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.987565 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-offline-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.987668 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.987608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-ui-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:53.993858 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:53.993838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wltw2\" (UniqueName: \"kubernetes.io/projected/839ba787-1aaf-4b21-9cab-007c2b53f42e-kube-api-access-wltw2\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:54.488386 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:54.488351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-online-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:54.490589 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:54.490558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-online-tls\") pod \"feast-simple-feast-remote-setup-6599699bfc-7fmv9\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:54.698302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:54.698270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:45:54.836744 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:54.836718 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9"] Jul 22 11:45:54.838817 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:45:54.838790 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod839ba787_1aaf_4b21_9cab_007c2b53f42e.slice/crio-1f09dc573dee96b5c2122ec5a0ceb287b550de99796b532f16241dfb43b6b75d WatchSource:0}: Error finding container 1f09dc573dee96b5c2122ec5a0ceb287b550de99796b532f16241dfb43b6b75d: Status 404 returned error can't find the container with id 1f09dc573dee96b5c2122ec5a0ceb287b550de99796b532f16241dfb43b6b75d Jul 22 11:45:54.860183 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:54.860160 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerStarted","Data":"1f09dc573dee96b5c2122ec5a0ceb287b550de99796b532f16241dfb43b6b75d"} Jul 22 11:45:55.866284 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:55.866249 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerStarted","Data":"2bfd40dd77de445af2571b2f2c2b51a06f46fa4c4e2522788571a8b58d1d78fe"} Jul 22 11:45:59.886559 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:59.886530 2579 generic.go:356] "Generic (PLEG): container finished" podID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerID="2bfd40dd77de445af2571b2f2c2b51a06f46fa4c4e2522788571a8b58d1d78fe" exitCode=0 Jul 22 11:45:59.886907 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:45:59.886606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerDied","Data":"2bfd40dd77de445af2571b2f2c2b51a06f46fa4c4e2522788571a8b58d1d78fe"} Jul 22 11:46:00.892300 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:00.892259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerStarted","Data":"26015d98439909863b57480a94894f723348523f8747114627c2a45cea2bdfab"} Jul 22 11:46:04.134427 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:04.134381 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:46:04.134876 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:04.134688 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:46:05.917842 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:05.917810 2579 generic.go:356] "Generic (PLEG): container finished" podID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerID="26015d98439909863b57480a94894f723348523f8747114627c2a45cea2bdfab" exitCode=0 Jul 22 11:46:05.918220 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:05.917890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerDied","Data":"26015d98439909863b57480a94894f723348523f8747114627c2a45cea2bdfab"} Jul 22 11:46:05.921653 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:05.921632 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 22 11:46:06.926799 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:06.926754 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerStarted","Data":"7d84b0b65ee3604aee49ddd23e7e625afa762bd6aa8a974475b98159c9fb351a"} Jul 22 11:46:06.927283 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:06.926817 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerStarted","Data":"7bc6d9bc68f6f47bed4d5016ad84bc10de7e56f26795f496a07d38e3f731037c"} Jul 22 11:46:06.927283 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:06.926837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerStarted","Data":"f3d58564b5251ab17013eb17bd2370821b352b8f85af2d46d9cf776c9e871a8d"} Jul 22 11:46:06.952173 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:06.952116 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podStartSLOduration=13.952096261 podStartE2EDuration="13.952096261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:46:06.950308158 +0000 UTC m=+603.249283164" watchObservedRunningTime="2026-07-22 11:46:06.952096261 +0000 UTC m=+603.251071219" Jul 22 11:46:09.698961 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:09.698923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:09.699365 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:09.698983 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:09.699365 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:09.699000 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:09.700766 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:09.700708 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Jul 22 11:46:09.700896 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:09.700716 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused" Jul 22 11:46:09.700896 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:09.700728 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jul 22 11:46:12.699289 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:12.699244 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:12.699767 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:12.699431 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:12.699821 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:12.699776 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:12.699821 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:12.699812 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:12.700136 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:12.700117 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:12.700286 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:12.700268 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:12.706317 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:12.706293 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:12.965825 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:12.965757 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:46:12.970022 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:46:12.969998 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:47:15.171971 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:15.171937 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9"] Jul 22 11:47:15.174564 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:15.172258 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="online" containerID="cri-o://f3d58564b5251ab17013eb17bd2370821b352b8f85af2d46d9cf776c9e871a8d" gracePeriod=30 Jul 22 11:47:15.174564 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:15.172330 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="offline" containerID="cri-o://7bc6d9bc68f6f47bed4d5016ad84bc10de7e56f26795f496a07d38e3f731037c" gracePeriod=30 Jul 22 11:47:15.174564 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:15.172330 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="ui" containerID="cri-o://7d84b0b65ee3604aee49ddd23e7e625afa762bd6aa8a974475b98159c9fb351a" gracePeriod=30 Jul 22 11:47:15.287341 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:15.287309 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n"] Jul 22 11:47:15.287935 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:15.287533 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="registry" containerID="cri-o://26594f7e16763a619c3212c262d06299f975b359870ef9b6b91ec5c039cd7e89" gracePeriod=30 Jul 22 11:47:15.287935 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:15.287616 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="ui" containerID="cri-o://3fa0f2fe2edd59d8afcc013f0892ab9d04ba0249c65e00e8c1efea9af031f908" gracePeriod=30 Jul 22 11:47:15.287935 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:15.287614 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="offline" containerID="cri-o://9a01195e4ba0dd42ffbc75955b2db7479785c6c37cc426b38b21be0492d4001f" gracePeriod=30 Jul 22 11:47:15.287935 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:15.287614 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="online" containerID="cri-o://0ac3860f799a2fbf62c2af7cdf4a9d696fefdbd077981aaf032b152d38e64efd" gracePeriod=30 Jul 22 11:47:16.254232 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:16.254107 2579 generic.go:356] "Generic (PLEG): container finished" podID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerID="7d84b0b65ee3604aee49ddd23e7e625afa762bd6aa8a974475b98159c9fb351a" exitCode=0 Jul 22 11:47:16.254232 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:16.254153 2579 generic.go:356] "Generic (PLEG): container finished" podID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerID="f3d58564b5251ab17013eb17bd2370821b352b8f85af2d46d9cf776c9e871a8d" exitCode=0 Jul 22 11:47:16.254232 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:16.254159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerDied","Data":"7d84b0b65ee3604aee49ddd23e7e625afa762bd6aa8a974475b98159c9fb351a"} Jul 22 11:47:16.254734 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:16.254247 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerDied","Data":"f3d58564b5251ab17013eb17bd2370821b352b8f85af2d46d9cf776c9e871a8d"} Jul 22 11:47:16.257158 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:16.257135 2579 generic.go:356] "Generic (PLEG): container finished" podID="28070ec7-6e93-4350-915e-3760778eef55" containerID="3fa0f2fe2edd59d8afcc013f0892ab9d04ba0249c65e00e8c1efea9af031f908" exitCode=0 Jul 22 11:47:16.257158 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:16.257153 2579 generic.go:356] "Generic (PLEG): container finished" podID="28070ec7-6e93-4350-915e-3760778eef55" containerID="0ac3860f799a2fbf62c2af7cdf4a9d696fefdbd077981aaf032b152d38e64efd" exitCode=0 Jul 22 11:47:16.257319 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:16.257223 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerDied","Data":"3fa0f2fe2edd59d8afcc013f0892ab9d04ba0249c65e00e8c1efea9af031f908"} Jul 22 11:47:16.257319 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:16.257256 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerDied","Data":"0ac3860f799a2fbf62c2af7cdf4a9d696fefdbd077981aaf032b152d38e64efd"} Jul 22 11:47:21.560752 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:21.560713 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Jul 22 11:47:21.561253 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:21.560728 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Jul 22 11:47:22.700072 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:22.700027 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Jul 22 11:47:22.966076 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:22.965982 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jul 22 11:47:31.561494 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:31.561445 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Jul 22 11:47:31.561494 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:31.561465 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Jul 22 11:47:32.699977 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:32.699934 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Jul 22 11:47:32.966685 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:32.966591 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jul 22 11:47:41.560958 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:41.560908 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Jul 22 11:47:41.561443 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:41.560912 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Jul 22 11:47:41.561443 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:41.561033 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:47:41.561443 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:41.561189 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:47:42.700285 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:42.700233 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Jul 22 11:47:42.700715 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:42.700386 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:47:42.965850 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:42.965753 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jul 22 11:47:42.966001 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:42.965876 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:47:45.393869 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.393838 2579 generic.go:356] "Generic (PLEG): container finished" podID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerID="7bc6d9bc68f6f47bed4d5016ad84bc10de7e56f26795f496a07d38e3f731037c" exitCode=137 Jul 22 11:47:45.394323 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.393912 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerDied","Data":"7bc6d9bc68f6f47bed4d5016ad84bc10de7e56f26795f496a07d38e3f731037c"} Jul 22 11:47:45.394323 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.393956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" event={"ID":"839ba787-1aaf-4b21-9cab-007c2b53f42e","Type":"ContainerDied","Data":"1f09dc573dee96b5c2122ec5a0ceb287b550de99796b532f16241dfb43b6b75d"} Jul 22 11:47:45.394323 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.393971 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f09dc573dee96b5c2122ec5a0ceb287b550de99796b532f16241dfb43b6b75d" Jul 22 11:47:45.397132 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.397112 2579 generic.go:356] "Generic (PLEG): container finished" podID="28070ec7-6e93-4350-915e-3760778eef55" containerID="9a01195e4ba0dd42ffbc75955b2db7479785c6c37cc426b38b21be0492d4001f" exitCode=137 Jul 22 11:47:45.397132 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.397128 2579 generic.go:356] "Generic (PLEG): container finished" podID="28070ec7-6e93-4350-915e-3760778eef55" containerID="26594f7e16763a619c3212c262d06299f975b359870ef9b6b91ec5c039cd7e89" exitCode=137 Jul 22 11:47:45.397328 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.397179 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerDied","Data":"9a01195e4ba0dd42ffbc75955b2db7479785c6c37cc426b38b21be0492d4001f"} Jul 22 11:47:45.397328 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.397227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerDied","Data":"26594f7e16763a619c3212c262d06299f975b359870ef9b6b91ec5c039cd7e89"} Jul 22 11:47:45.410374 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.410357 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:47:45.482433 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.482351 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/839ba787-1aaf-4b21-9cab-007c2b53f42e-registry-tls\") pod \"839ba787-1aaf-4b21-9cab-007c2b53f42e\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " Jul 22 11:47:45.482433 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.482401 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/839ba787-1aaf-4b21-9cab-007c2b53f42e-feast-data\") pod \"839ba787-1aaf-4b21-9cab-007c2b53f42e\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " Jul 22 11:47:45.482433 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.482430 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-online-tls\") pod \"839ba787-1aaf-4b21-9cab-007c2b53f42e\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " Jul 22 11:47:45.482627 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.482591 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-offline-tls\") pod \"839ba787-1aaf-4b21-9cab-007c2b53f42e\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " Jul 22 11:47:45.482667 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.482659 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wltw2\" (UniqueName: \"kubernetes.io/projected/839ba787-1aaf-4b21-9cab-007c2b53f42e-kube-api-access-wltw2\") pod \"839ba787-1aaf-4b21-9cab-007c2b53f42e\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " Jul 22 11:47:45.482722 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.482683 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-ui-tls\") pod \"839ba787-1aaf-4b21-9cab-007c2b53f42e\" (UID: \"839ba787-1aaf-4b21-9cab-007c2b53f42e\") " Jul 22 11:47:45.482827 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.482772 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839ba787-1aaf-4b21-9cab-007c2b53f42e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "839ba787-1aaf-4b21-9cab-007c2b53f42e" (UID: "839ba787-1aaf-4b21-9cab-007c2b53f42e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:47:45.482968 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.482932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839ba787-1aaf-4b21-9cab-007c2b53f42e-feast-data" (OuterVolumeSpecName: "feast-data") pod "839ba787-1aaf-4b21-9cab-007c2b53f42e" (UID: "839ba787-1aaf-4b21-9cab-007c2b53f42e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:47:45.483038 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.483003 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/839ba787-1aaf-4b21-9cab-007c2b53f42e-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:45.484810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.484773 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "839ba787-1aaf-4b21-9cab-007c2b53f42e" (UID: "839ba787-1aaf-4b21-9cab-007c2b53f42e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:47:45.484810 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.484772 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839ba787-1aaf-4b21-9cab-007c2b53f42e-kube-api-access-wltw2" (OuterVolumeSpecName: "kube-api-access-wltw2") pod "839ba787-1aaf-4b21-9cab-007c2b53f42e" (UID: "839ba787-1aaf-4b21-9cab-007c2b53f42e"). InnerVolumeSpecName "kube-api-access-wltw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:47:45.484936 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.484817 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "839ba787-1aaf-4b21-9cab-007c2b53f42e" (UID: "839ba787-1aaf-4b21-9cab-007c2b53f42e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:47:45.484936 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.484848 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-online-tls" (OuterVolumeSpecName: "online-tls") pod "839ba787-1aaf-4b21-9cab-007c2b53f42e" (UID: "839ba787-1aaf-4b21-9cab-007c2b53f42e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:47:45.583839 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.583792 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/839ba787-1aaf-4b21-9cab-007c2b53f42e-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:45.583839 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.583833 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:45.583839 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.583842 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-offline-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:45.583839 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.583851 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wltw2\" (UniqueName: \"kubernetes.io/projected/839ba787-1aaf-4b21-9cab-007c2b53f42e-kube-api-access-wltw2\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:45.584106 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.583862 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/839ba787-1aaf-4b21-9cab-007c2b53f42e-ui-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:45.937346 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.937319 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:47:45.987242 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.987178 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8kpj\" (UniqueName: \"kubernetes.io/projected/28070ec7-6e93-4350-915e-3760778eef55-kube-api-access-p8kpj\") pod \"28070ec7-6e93-4350-915e-3760778eef55\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " Jul 22 11:47:45.987242 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.987246 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-online-tls\") pod \"28070ec7-6e93-4350-915e-3760778eef55\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " Jul 22 11:47:45.987459 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.987278 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-ui-tls\") pod \"28070ec7-6e93-4350-915e-3760778eef55\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " Jul 22 11:47:45.987459 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.987303 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-offline-tls\") pod \"28070ec7-6e93-4350-915e-3760778eef55\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " Jul 22 11:47:45.987459 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.987380 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-registry-tls\") pod \"28070ec7-6e93-4350-915e-3760778eef55\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " Jul 22 11:47:45.987459 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.987432 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/28070ec7-6e93-4350-915e-3760778eef55-feast-data\") pod \"28070ec7-6e93-4350-915e-3760778eef55\" (UID: \"28070ec7-6e93-4350-915e-3760778eef55\") " Jul 22 11:47:45.988032 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.987997 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28070ec7-6e93-4350-915e-3760778eef55-feast-data" (OuterVolumeSpecName: "feast-data") pod "28070ec7-6e93-4350-915e-3760778eef55" (UID: "28070ec7-6e93-4350-915e-3760778eef55"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:47:45.989365 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.989334 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "28070ec7-6e93-4350-915e-3760778eef55" (UID: "28070ec7-6e93-4350-915e-3760778eef55"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:47:45.989801 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.989779 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-online-tls" (OuterVolumeSpecName: "online-tls") pod "28070ec7-6e93-4350-915e-3760778eef55" (UID: "28070ec7-6e93-4350-915e-3760778eef55"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:47:45.989865 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.989798 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28070ec7-6e93-4350-915e-3760778eef55-kube-api-access-p8kpj" (OuterVolumeSpecName: "kube-api-access-p8kpj") pod "28070ec7-6e93-4350-915e-3760778eef55" (UID: "28070ec7-6e93-4350-915e-3760778eef55"). InnerVolumeSpecName "kube-api-access-p8kpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:47:45.989865 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.989809 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "28070ec7-6e93-4350-915e-3760778eef55" (UID: "28070ec7-6e93-4350-915e-3760778eef55"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:47:45.989865 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:45.989821 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "28070ec7-6e93-4350-915e-3760778eef55" (UID: "28070ec7-6e93-4350-915e-3760778eef55"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:47:46.088529 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.088464 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:46.088529 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.088488 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/28070ec7-6e93-4350-915e-3760778eef55-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:46.088529 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.088498 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8kpj\" (UniqueName: \"kubernetes.io/projected/28070ec7-6e93-4350-915e-3760778eef55-kube-api-access-p8kpj\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:46.088529 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.088506 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:46.088529 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.088516 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-ui-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:46.088529 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.088524 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28070ec7-6e93-4350-915e-3760778eef55-offline-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:47:46.404547 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.404518 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" Jul 22 11:47:46.404547 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.404527 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n" event={"ID":"28070ec7-6e93-4350-915e-3760778eef55","Type":"ContainerDied","Data":"0d3441666f3cc86a23e180c6bac128d5738356cddf85356713843d71aba22508"} Jul 22 11:47:46.405034 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.404583 2579 scope.go:117] "RemoveContainer" containerID="3fa0f2fe2edd59d8afcc013f0892ab9d04ba0249c65e00e8c1efea9af031f908" Jul 22 11:47:46.405034 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.404780 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9" Jul 22 11:47:46.413882 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.413865 2579 scope.go:117] "RemoveContainer" containerID="9a01195e4ba0dd42ffbc75955b2db7479785c6c37cc426b38b21be0492d4001f" Jul 22 11:47:46.421547 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.421528 2579 scope.go:117] "RemoveContainer" containerID="0ac3860f799a2fbf62c2af7cdf4a9d696fefdbd077981aaf032b152d38e64efd" Jul 22 11:47:46.425771 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.425748 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n"] Jul 22 11:47:46.432574 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.430601 2579 scope.go:117] "RemoveContainer" containerID="26594f7e16763a619c3212c262d06299f975b359870ef9b6b91ec5c039cd7e89" Jul 22 11:47:46.432574 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.430988 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c874b989b-dkc2n"] Jul 22 11:47:46.442720 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.442695 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9"] Jul 22 11:47:46.443642 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.443620 2579 scope.go:117] "RemoveContainer" containerID="7ea9fa750e52b0b8b750b2bed8c21ab059702e6d8ebe98499305e2e8e7c1ac2a" Jul 22 11:47:46.446832 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.446810 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-6599699bfc-7fmv9"] Jul 22 11:47:46.451539 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:46.451526 2579 scope.go:117] "RemoveContainer" containerID="4c155dffc3b8635356265ac695a3ead6dfe005cb4c6c3b9631d6e7e35219bd8c" Jul 22 11:47:48.205947 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:48.205916 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28070ec7-6e93-4350-915e-3760778eef55" path="/var/lib/kubelet/pods/28070ec7-6e93-4350-915e-3760778eef55/volumes" Jul 22 11:47:48.206538 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:48.206523 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" path="/var/lib/kubelet/pods/839ba787-1aaf-4b21-9cab-007c2b53f42e/volumes" Jul 22 11:47:49.289917 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.289888 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-6wkcf"] Jul 22 11:47:49.290302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290239 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="online" Jul 22 11:47:49.290302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290252 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="online" Jul 22 11:47:49.290302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290261 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="ui" Jul 22 11:47:49.290302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290267 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="ui" Jul 22 11:47:49.290302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290281 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="registry" Jul 22 11:47:49.290302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290287 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="registry" Jul 22 11:47:49.290302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290298 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="feast-apply" Jul 22 11:47:49.290302 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290305 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="feast-apply" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290315 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="feast-init" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290321 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="feast-init" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290326 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="online" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290333 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="online" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290339 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="offline" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290344 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="offline" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290351 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="offline" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290356 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="offline" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290362 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="feast-apply" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290368 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="feast-apply" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290376 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="ui" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290382 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="ui" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290395 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="feast-init" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290400 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="feast-init" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290451 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="ui" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290458 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="online" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290467 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="offline" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290476 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="online" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290481 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="839ba787-1aaf-4b21-9cab-007c2b53f42e" containerName="offline" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290489 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="ui" Jul 22 11:47:49.290541 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.290496 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="28070ec7-6e93-4350-915e-3760778eef55" containerName="registry" Jul 22 11:47:49.295113 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.295092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-6wkcf" Jul 22 11:47:49.297877 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.297859 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 22 11:47:49.298978 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.298955 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-fpp95\"" Jul 22 11:47:49.299080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.298974 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 22 11:47:49.299080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.298955 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jul 22 11:47:49.302092 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.301915 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-6wkcf"] Jul 22 11:47:49.411653 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.411627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4mc\" (UniqueName: \"kubernetes.io/projected/c41c9528-6bb8-4732-b9f7-02188024ce43-kube-api-access-zp4mc\") pod \"postgres-864846d4f5-6wkcf\" (UID: \"c41c9528-6bb8-4732-b9f7-02188024ce43\") " pod="test-ns-feast/postgres-864846d4f5-6wkcf" Jul 22 11:47:49.411811 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.411666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c41c9528-6bb8-4732-b9f7-02188024ce43-postgresdata\") pod \"postgres-864846d4f5-6wkcf\" (UID: \"c41c9528-6bb8-4732-b9f7-02188024ce43\") " pod="test-ns-feast/postgres-864846d4f5-6wkcf" Jul 22 11:47:49.512670 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.512638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c41c9528-6bb8-4732-b9f7-02188024ce43-postgresdata\") pod \"postgres-864846d4f5-6wkcf\" (UID: \"c41c9528-6bb8-4732-b9f7-02188024ce43\") " pod="test-ns-feast/postgres-864846d4f5-6wkcf" Jul 22 11:47:49.512800 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.512734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4mc\" (UniqueName: \"kubernetes.io/projected/c41c9528-6bb8-4732-b9f7-02188024ce43-kube-api-access-zp4mc\") pod \"postgres-864846d4f5-6wkcf\" (UID: \"c41c9528-6bb8-4732-b9f7-02188024ce43\") " pod="test-ns-feast/postgres-864846d4f5-6wkcf" Jul 22 11:47:49.513080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.513060 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c41c9528-6bb8-4732-b9f7-02188024ce43-postgresdata\") pod \"postgres-864846d4f5-6wkcf\" (UID: \"c41c9528-6bb8-4732-b9f7-02188024ce43\") " pod="test-ns-feast/postgres-864846d4f5-6wkcf" Jul 22 11:47:49.522118 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.522077 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4mc\" (UniqueName: \"kubernetes.io/projected/c41c9528-6bb8-4732-b9f7-02188024ce43-kube-api-access-zp4mc\") pod \"postgres-864846d4f5-6wkcf\" (UID: \"c41c9528-6bb8-4732-b9f7-02188024ce43\") " pod="test-ns-feast/postgres-864846d4f5-6wkcf" Jul 22 11:47:49.612163 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.612102 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-6wkcf" Jul 22 11:47:49.741728 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.741703 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-6wkcf"] Jul 22 11:47:49.746305 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:47:49.746261 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc41c9528_6bb8_4732_b9f7_02188024ce43.slice/crio-b907f3e377b886b147a18c31cc9ce1889baaa1b91d28662c9a73c9641a1fbfcd WatchSource:0}: Error finding container b907f3e377b886b147a18c31cc9ce1889baaa1b91d28662c9a73c9641a1fbfcd: Status 404 returned error can't find the container with id b907f3e377b886b147a18c31cc9ce1889baaa1b91d28662c9a73c9641a1fbfcd Jul 22 11:47:49.748129 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:49.748111 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:47:50.424388 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:50.424358 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-6wkcf" event={"ID":"c41c9528-6bb8-4732-b9f7-02188024ce43","Type":"ContainerStarted","Data":"b907f3e377b886b147a18c31cc9ce1889baaa1b91d28662c9a73c9641a1fbfcd"} Jul 22 11:47:55.449129 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:55.449085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-6wkcf" event={"ID":"c41c9528-6bb8-4732-b9f7-02188024ce43","Type":"ContainerStarted","Data":"d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd"} Jul 22 11:47:55.466127 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:55.466064 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-6wkcf" podStartSLOduration=1.4719186180000001 podStartE2EDuration="6.466050975s" totalImagesPullingTime="4.994132357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:47:55.464986036 +0000 UTC m=+711.763960997" watchObservedRunningTime="2026-07-22 11:47:55.466050975 +0000 UTC m=+711.765025932" Jul 22 11:47:59.943152 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:59.943070 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-994c876c6-q76zr"] Jul 22 11:47:59.946724 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:59.946706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:47:59.950743 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:59.950712 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jul 22 11:47:59.950878 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:59.950750 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jul 22 11:47:59.950878 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:59.950766 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jul 22 11:47:59.951084 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:59.951069 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-5zfzw\"" Jul 22 11:47:59.951250 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:59.951236 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jul 22 11:47:59.965909 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:47:59.965887 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-994c876c6-q76zr"] Jul 22 11:48:00.001757 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.001727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-online-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.001871 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.001808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-offline-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.001871 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.001855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqhlw\" (UniqueName: \"kubernetes.io/projected/5766bf12-542b-458e-94a8-a564875de3b8-kube-api-access-rqhlw\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.001945 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.001881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5766bf12-542b-458e-94a8-a564875de3b8-feast-data\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.001945 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.001913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-registry-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.102418 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.102380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-online-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.102581 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.102429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-offline-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.102581 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.102467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqhlw\" (UniqueName: \"kubernetes.io/projected/5766bf12-542b-458e-94a8-a564875de3b8-kube-api-access-rqhlw\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.102581 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.102487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5766bf12-542b-458e-94a8-a564875de3b8-feast-data\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.102581 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.102533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-registry-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.102831 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:48:00.102635 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Jul 22 11:48:00.102831 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:48:00.102740 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-offline-tls podName:5766bf12-542b-458e-94a8-a564875de3b8 nodeName:}" failed. No retries permitted until 2026-07-22 11:48:00.60271861 +0000 UTC m=+716.901693559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-offline-tls") pod "feast-credit-scoring-994c876c6-q76zr" (UID: "5766bf12-542b-458e-94a8-a564875de3b8") : secret "feast-credit-scoring-offline-tls" not found Jul 22 11:48:00.102831 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:48:00.102643 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Jul 22 11:48:00.102831 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:48:00.102789 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-registry-tls podName:5766bf12-542b-458e-94a8-a564875de3b8 nodeName:}" failed. No retries permitted until 2026-07-22 11:48:00.602771877 +0000 UTC m=+716.901746813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-registry-tls") pod "feast-credit-scoring-994c876c6-q76zr" (UID: "5766bf12-542b-458e-94a8-a564875de3b8") : secret "feast-credit-scoring-registry-tls" not found Jul 22 11:48:00.103021 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.102882 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5766bf12-542b-458e-94a8-a564875de3b8-feast-data\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.109783 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.109753 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-online-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.116699 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.116671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqhlw\" (UniqueName: \"kubernetes.io/projected/5766bf12-542b-458e-94a8-a564875de3b8-kube-api-access-rqhlw\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.607479 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.607444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-offline-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.607641 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.607504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-registry-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.610056 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.610024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-offline-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.610204 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.610033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-registry-tls\") pod \"feast-credit-scoring-994c876c6-q76zr\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:00.855420 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:00.855383 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:01.029091 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:01.029051 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-994c876c6-q76zr"] Jul 22 11:48:01.032627 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:48:01.032600 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5766bf12_542b_458e_94a8_a564875de3b8.slice/crio-128e9c6228b6bbec6eb977a633acb7295134319d5f94b6f17ce942991adcb59f WatchSource:0}: Error finding container 128e9c6228b6bbec6eb977a633acb7295134319d5f94b6f17ce942991adcb59f: Status 404 returned error can't find the container with id 128e9c6228b6bbec6eb977a633acb7295134319d5f94b6f17ce942991adcb59f Jul 22 11:48:01.477742 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:01.477706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerStarted","Data":"563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7"} Jul 22 11:48:01.477894 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:01.477764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerStarted","Data":"128e9c6228b6bbec6eb977a633acb7295134319d5f94b6f17ce942991adcb59f"} Jul 22 11:48:02.483343 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:02.483312 2579 generic.go:356] "Generic (PLEG): container finished" podID="5766bf12-542b-458e-94a8-a564875de3b8" containerID="563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7" exitCode=0 Jul 22 11:48:02.483717 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:02.483376 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerDied","Data":"563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7"} Jul 22 11:48:03.490642 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:03.490604 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerStarted","Data":"6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a"} Jul 22 11:48:09.519665 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:09.519626 2579 generic.go:356] "Generic (PLEG): container finished" podID="5766bf12-542b-458e-94a8-a564875de3b8" containerID="6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a" exitCode=0 Jul 22 11:48:09.520065 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:09.519669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerDied","Data":"6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a"} Jul 22 11:48:09.523308 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:09.523288 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jul 22 11:48:10.530501 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:10.530459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerStarted","Data":"5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02"} Jul 22 11:48:10.530989 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:10.530535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerStarted","Data":"d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5"} Jul 22 11:48:10.530989 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:10.530563 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerStarted","Data":"85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb"} Jul 22 11:48:10.556616 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:10.556549 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podStartSLOduration=11.556525173 podStartE2EDuration="11.556525173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:48:10.555128523 +0000 UTC m=+726.854103530" watchObservedRunningTime="2026-07-22 11:48:10.556525173 +0000 UTC m=+726.855500132" Jul 22 11:48:10.856452 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:10.856360 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:10.856452 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:10.856425 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:10.856452 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:10.856443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:12.855527 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:12.855465 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:12.855927 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:12.855545 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:12.855927 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:12.855573 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:12.857788 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:12.857745 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.32:6571: connect: connection refused" Jul 22 11:48:12.857937 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:12.857784 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.32:8816: connect: connection refused" Jul 22 11:48:12.857937 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:12.857818 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Jul 22 11:48:15.856483 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:15.856437 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.32:6571: connect: connection refused" Jul 22 11:48:15.857449 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:15.857419 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:15.858352 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:15.858327 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:15.862047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:15.862026 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:16.566868 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:16.566836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:18.856660 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:18.856609 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.32:6571: connect: connection refused" Jul 22 11:48:21.856940 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:21.856912 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:48:21.857776 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:48:21.857752 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:49:25.140409 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.140377 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-994c876c6-q76zr"] Jul 22 11:49:25.140913 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.140587 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="registry" containerID="cri-o://85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb" gracePeriod=30 Jul 22 11:49:25.140913 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.140643 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="online" containerID="cri-o://d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5" gracePeriod=30 Jul 22 11:49:25.140913 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.140653 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="offline" containerID="cri-o://5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02" gracePeriod=30 Jul 22 11:49:25.162323 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.162296 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-6wkcf"] Jul 22 11:49:25.162475 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.162446 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-6wkcf" podUID="c41c9528-6bb8-4732-b9f7-02188024ce43" containerName="postgres" containerID="cri-o://d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd" gracePeriod=30 Jul 22 11:49:25.401027 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.400966 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-6wkcf" Jul 22 11:49:25.431500 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.431213 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c41c9528-6bb8-4732-b9f7-02188024ce43-postgresdata\") pod \"c41c9528-6bb8-4732-b9f7-02188024ce43\" (UID: \"c41c9528-6bb8-4732-b9f7-02188024ce43\") " Jul 22 11:49:25.431500 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.431291 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp4mc\" (UniqueName: \"kubernetes.io/projected/c41c9528-6bb8-4732-b9f7-02188024ce43-kube-api-access-zp4mc\") pod \"c41c9528-6bb8-4732-b9f7-02188024ce43\" (UID: \"c41c9528-6bb8-4732-b9f7-02188024ce43\") " Jul 22 11:49:25.433965 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.433926 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41c9528-6bb8-4732-b9f7-02188024ce43-kube-api-access-zp4mc" (OuterVolumeSpecName: "kube-api-access-zp4mc") pod "c41c9528-6bb8-4732-b9f7-02188024ce43" (UID: "c41c9528-6bb8-4732-b9f7-02188024ce43"). InnerVolumeSpecName "kube-api-access-zp4mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:49:25.460798 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.460768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41c9528-6bb8-4732-b9f7-02188024ce43-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "c41c9528-6bb8-4732-b9f7-02188024ce43" (UID: "c41c9528-6bb8-4732-b9f7-02188024ce43"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:49:25.532007 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.531982 2579 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c41c9528-6bb8-4732-b9f7-02188024ce43-postgresdata\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:49:25.532007 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.532004 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4mc\" (UniqueName: \"kubernetes.io/projected/c41c9528-6bb8-4732-b9f7-02188024ce43-kube-api-access-zp4mc\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:49:25.891544 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.891513 2579 generic.go:356] "Generic (PLEG): container finished" podID="c41c9528-6bb8-4732-b9f7-02188024ce43" containerID="d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd" exitCode=0 Jul 22 11:49:25.891641 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.891549 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-6wkcf" Jul 22 11:49:25.891641 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.891593 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-6wkcf" event={"ID":"c41c9528-6bb8-4732-b9f7-02188024ce43","Type":"ContainerDied","Data":"d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd"} Jul 22 11:49:25.891641 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.891630 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-6wkcf" event={"ID":"c41c9528-6bb8-4732-b9f7-02188024ce43","Type":"ContainerDied","Data":"b907f3e377b886b147a18c31cc9ce1889baaa1b91d28662c9a73c9641a1fbfcd"} Jul 22 11:49:25.891740 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.891661 2579 scope.go:117] "RemoveContainer" containerID="d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd" Jul 22 11:49:25.911440 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.911415 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-6wkcf"] Jul 22 11:49:25.914983 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.914964 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-6wkcf"] Jul 22 11:49:25.951314 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.951293 2579 scope.go:117] "RemoveContainer" containerID="d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd" Jul 22 11:49:25.951559 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:49:25.951540 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd\": container with ID starting with d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd not found: ID does not exist" containerID="d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd" Jul 22 11:49:25.951614 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:25.951567 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd"} err="failed to get container status \"d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd\": rpc error: code = NotFound desc = could not find container \"d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd\": container with ID starting with d94a264a495975869822ee7e89e65393654a5256105454529bc17d56a9a496cd not found: ID does not exist" Jul 22 11:49:26.205268 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:26.205246 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c41c9528-6bb8-4732-b9f7-02188024ce43" path="/var/lib/kubelet/pods/c41c9528-6bb8-4732-b9f7-02188024ce43/volumes" Jul 22 11:49:26.563794 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:26.563714 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Jul 22 11:49:26.897930 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:26.897906 2579 generic.go:356] "Generic (PLEG): container finished" podID="5766bf12-542b-458e-94a8-a564875de3b8" containerID="d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5" exitCode=0 Jul 22 11:49:26.898058 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:26.897977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerDied","Data":"d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5"} Jul 22 11:49:30.652750 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.652714 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b9c7758c4-4mp8m"] Jul 22 11:49:30.653540 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.653214 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c41c9528-6bb8-4732-b9f7-02188024ce43" containerName="postgres" Jul 22 11:49:30.653540 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.653233 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c41c9528-6bb8-4732-b9f7-02188024ce43" containerName="postgres" Jul 22 11:49:30.653540 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.653340 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c41c9528-6bb8-4732-b9f7-02188024ce43" containerName="postgres" Jul 22 11:49:30.656576 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.656554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.665793 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.665771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-console-serving-cert\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.665893 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.665805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnv7\" (UniqueName: \"kubernetes.io/projected/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-kube-api-access-fpnv7\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.665893 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.665834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-service-ca\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.665893 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.665871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-console-config\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.666007 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.665899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-trusted-ca-bundle\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.666007 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.665967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-oauth-serving-cert\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.666007 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.666004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-console-oauth-config\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.670428 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.670404 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9c7758c4-4mp8m"] Jul 22 11:49:30.767257 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.767227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-trusted-ca-bundle\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.767384 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.767262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-oauth-serving-cert\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.767384 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.767288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-console-oauth-config\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.767384 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.767325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-console-serving-cert\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.767529 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.767441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnv7\" (UniqueName: \"kubernetes.io/projected/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-kube-api-access-fpnv7\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.767529 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.767496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-service-ca\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.767529 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.767528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-console-config\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.768041 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.768019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-trusted-ca-bundle\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.768140 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.768029 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-oauth-serving-cert\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.768252 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.768233 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-console-config\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.768252 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.768244 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-service-ca\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.769648 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.769629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-console-serving-cert\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.769866 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.769841 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-console-oauth-config\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.775640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.775616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnv7\" (UniqueName: \"kubernetes.io/projected/21f2fdfc-9b45-49cb-bdb5-860ddbb362a2-kube-api-access-fpnv7\") pod \"console-5b9c7758c4-4mp8m\" (UID: \"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2\") " pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:30.965996 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:30.965944 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:31.096365 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:31.096226 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9c7758c4-4mp8m"] Jul 22 11:49:31.099052 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:49:31.099021 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f2fdfc_9b45_49cb_bdb5_860ddbb362a2.slice/crio-2dbb51f92dd1f0eadc7b28dd370721426d8aa3a5531fba719d80b33e876b3d76 WatchSource:0}: Error finding container 2dbb51f92dd1f0eadc7b28dd370721426d8aa3a5531fba719d80b33e876b3d76: Status 404 returned error can't find the container with id 2dbb51f92dd1f0eadc7b28dd370721426d8aa3a5531fba719d80b33e876b3d76 Jul 22 11:49:31.921013 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:31.920984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9c7758c4-4mp8m" event={"ID":"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2","Type":"ContainerStarted","Data":"2423933f49be611f67aa47c83b57e097c742b29e55842c1869d06a253e9f0738"} Jul 22 11:49:31.921383 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:31.921030 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9c7758c4-4mp8m" event={"ID":"21f2fdfc-9b45-49cb-bdb5-860ddbb362a2","Type":"ContainerStarted","Data":"2dbb51f92dd1f0eadc7b28dd370721426d8aa3a5531fba719d80b33e876b3d76"} Jul 22 11:49:31.941394 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:31.941345 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b9c7758c4-4mp8m" podStartSLOduration=1.941331629 podStartE2EDuration="1.941331629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:49:31.939983794 +0000 UTC m=+808.238958751" watchObservedRunningTime="2026-07-22 11:49:31.941331629 +0000 UTC m=+808.240306586" Jul 22 11:49:36.564371 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:36.564331 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Jul 22 11:49:40.966811 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:40.966779 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:40.967179 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:40.966855 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:40.971294 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:40.971268 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:41.965767 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:41.965739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b9c7758c4-4mp8m" Jul 22 11:49:42.017186 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:42.017155 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-695df56d48-vbxtc"] Jul 22 11:49:46.564218 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:46.564151 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.32:6567/health\": dial tcp 10.132.0.32:6567: connect: connection refused" Jul 22 11:49:46.564612 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:46.564309 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:49:55.778728 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.778702 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:49:55.858453 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.858427 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-offline-tls\") pod \"5766bf12-542b-458e-94a8-a564875de3b8\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " Jul 22 11:49:55.858613 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.858461 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-online-tls\") pod \"5766bf12-542b-458e-94a8-a564875de3b8\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " Jul 22 11:49:55.858613 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.858515 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-registry-tls\") pod \"5766bf12-542b-458e-94a8-a564875de3b8\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " Jul 22 11:49:55.858613 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.858542 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqhlw\" (UniqueName: \"kubernetes.io/projected/5766bf12-542b-458e-94a8-a564875de3b8-kube-api-access-rqhlw\") pod \"5766bf12-542b-458e-94a8-a564875de3b8\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " Jul 22 11:49:55.858613 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.858560 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5766bf12-542b-458e-94a8-a564875de3b8-feast-data\") pod \"5766bf12-542b-458e-94a8-a564875de3b8\" (UID: \"5766bf12-542b-458e-94a8-a564875de3b8\") " Jul 22 11:49:55.860547 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.860518 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-online-tls" (OuterVolumeSpecName: "online-tls") pod "5766bf12-542b-458e-94a8-a564875de3b8" (UID: "5766bf12-542b-458e-94a8-a564875de3b8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:49:55.860666 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.860563 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5766bf12-542b-458e-94a8-a564875de3b8" (UID: "5766bf12-542b-458e-94a8-a564875de3b8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:49:55.860666 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.860648 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5766bf12-542b-458e-94a8-a564875de3b8-kube-api-access-rqhlw" (OuterVolumeSpecName: "kube-api-access-rqhlw") pod "5766bf12-542b-458e-94a8-a564875de3b8" (UID: "5766bf12-542b-458e-94a8-a564875de3b8"). InnerVolumeSpecName "kube-api-access-rqhlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:49:55.860825 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.860808 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5766bf12-542b-458e-94a8-a564875de3b8" (UID: "5766bf12-542b-458e-94a8-a564875de3b8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:49:55.866947 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.866922 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5766bf12-542b-458e-94a8-a564875de3b8-feast-data" (OuterVolumeSpecName: "feast-data") pod "5766bf12-542b-458e-94a8-a564875de3b8" (UID: "5766bf12-542b-458e-94a8-a564875de3b8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:49:55.959082 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.959034 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:49:55.959082 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.959055 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqhlw\" (UniqueName: \"kubernetes.io/projected/5766bf12-542b-458e-94a8-a564875de3b8-kube-api-access-rqhlw\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:49:55.959082 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.959066 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5766bf12-542b-458e-94a8-a564875de3b8-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:49:55.959082 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.959075 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-offline-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:49:55.959337 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:55.959107 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5766bf12-542b-458e-94a8-a564875de3b8-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:49:56.034587 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.034565 2579 generic.go:356] "Generic (PLEG): container finished" podID="5766bf12-542b-458e-94a8-a564875de3b8" containerID="5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02" exitCode=137 Jul 22 11:49:56.034587 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.034583 2579 generic.go:356] "Generic (PLEG): container finished" podID="5766bf12-542b-458e-94a8-a564875de3b8" containerID="85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb" exitCode=137 Jul 22 11:49:56.034741 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.034617 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerDied","Data":"5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02"} Jul 22 11:49:56.034741 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.034642 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" Jul 22 11:49:56.034741 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.034674 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerDied","Data":"85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb"} Jul 22 11:49:56.034741 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.034693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-994c876c6-q76zr" event={"ID":"5766bf12-542b-458e-94a8-a564875de3b8","Type":"ContainerDied","Data":"128e9c6228b6bbec6eb977a633acb7295134319d5f94b6f17ce942991adcb59f"} Jul 22 11:49:56.034741 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.034713 2579 scope.go:117] "RemoveContainer" containerID="5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02" Jul 22 11:49:56.043728 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.043714 2579 scope.go:117] "RemoveContainer" containerID="d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5" Jul 22 11:49:56.050840 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.050826 2579 scope.go:117] "RemoveContainer" containerID="85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb" Jul 22 11:49:56.057529 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.057514 2579 scope.go:117] "RemoveContainer" containerID="6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a" Jul 22 11:49:56.060888 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.060868 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-994c876c6-q76zr"] Jul 22 11:49:56.064685 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.064660 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-994c876c6-q76zr"] Jul 22 11:49:56.065897 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.065882 2579 scope.go:117] "RemoveContainer" containerID="563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7" Jul 22 11:49:56.072824 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.072806 2579 scope.go:117] "RemoveContainer" containerID="5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02" Jul 22 11:49:56.073054 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:49:56.073034 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02\": container with ID starting with 5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02 not found: ID does not exist" containerID="5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02" Jul 22 11:49:56.073121 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.073065 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02"} err="failed to get container status \"5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02\": rpc error: code = NotFound desc = could not find container \"5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02\": container with ID starting with 5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02 not found: ID does not exist" Jul 22 11:49:56.073121 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.073088 2579 scope.go:117] "RemoveContainer" containerID="d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5" Jul 22 11:49:56.073350 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:49:56.073332 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5\": container with ID starting with d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5 not found: ID does not exist" containerID="d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5" Jul 22 11:49:56.073395 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.073355 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5"} err="failed to get container status \"d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5\": rpc error: code = NotFound desc = could not find container \"d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5\": container with ID starting with d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5 not found: ID does not exist" Jul 22 11:49:56.073395 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.073372 2579 scope.go:117] "RemoveContainer" containerID="85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb" Jul 22 11:49:56.073568 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:49:56.073550 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb\": container with ID starting with 85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb not found: ID does not exist" containerID="85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb" Jul 22 11:49:56.073604 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.073578 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb"} err="failed to get container status \"85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb\": rpc error: code = NotFound desc = could not find container \"85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb\": container with ID starting with 85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb not found: ID does not exist" Jul 22 11:49:56.073604 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.073594 2579 scope.go:117] "RemoveContainer" containerID="6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a" Jul 22 11:49:56.073789 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:49:56.073771 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a\": container with ID starting with 6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a not found: ID does not exist" containerID="6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a" Jul 22 11:49:56.073851 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.073800 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a"} err="failed to get container status \"6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a\": rpc error: code = NotFound desc = could not find container \"6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a\": container with ID starting with 6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a not found: ID does not exist" Jul 22 11:49:56.073851 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.073821 2579 scope.go:117] "RemoveContainer" containerID="563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7" Jul 22 11:49:56.074021 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:49:56.074006 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7\": container with ID starting with 563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7 not found: ID does not exist" containerID="563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7" Jul 22 11:49:56.074059 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.074024 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7"} err="failed to get container status \"563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7\": rpc error: code = NotFound desc = could not find container \"563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7\": container with ID starting with 563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7 not found: ID does not exist" Jul 22 11:49:56.074059 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.074035 2579 scope.go:117] "RemoveContainer" containerID="5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02" Jul 22 11:49:56.074263 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.074245 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02"} err="failed to get container status \"5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02\": rpc error: code = NotFound desc = could not find container \"5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02\": container with ID starting with 5a1dfdec0e9328a9254fdc26ce98dee8f9dee90e754ed50ca4551ef469c08f02 not found: ID does not exist" Jul 22 11:49:56.074323 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.074263 2579 scope.go:117] "RemoveContainer" containerID="d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5" Jul 22 11:49:56.074516 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.074498 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5"} err="failed to get container status \"d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5\": rpc error: code = NotFound desc = could not find container \"d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5\": container with ID starting with d0fe1a04aa5003cedcb6f0da03c9ab3ad5b7492d313aa2b41ae250bb607f7ea5 not found: ID does not exist" Jul 22 11:49:56.074563 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.074516 2579 scope.go:117] "RemoveContainer" containerID="85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb" Jul 22 11:49:56.074742 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.074726 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb"} err="failed to get container status \"85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb\": rpc error: code = NotFound desc = could not find container \"85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb\": container with ID starting with 85bcf087cd203d64c187f3d79133981e98f7ddab4c8492cc474e0144ffa504bb not found: ID does not exist" Jul 22 11:49:56.074780 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.074743 2579 scope.go:117] "RemoveContainer" containerID="6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a" Jul 22 11:49:56.074965 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.074946 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a"} err="failed to get container status \"6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a\": rpc error: code = NotFound desc = could not find container \"6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a\": container with ID starting with 6aad5e1686e6d34012a6490e11414a2e41a565113a415eca0ef0f36892445e3a not found: ID does not exist" Jul 22 11:49:56.075013 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.074965 2579 scope.go:117] "RemoveContainer" containerID="563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7" Jul 22 11:49:56.077876 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.075155 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7"} err="failed to get container status \"563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7\": rpc error: code = NotFound desc = could not find container \"563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7\": container with ID starting with 563fb3ac813130539581374918b94ec8cac25b7d15ebcf549d73d242d45088f7 not found: ID does not exist" Jul 22 11:49:56.205579 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:49:56.205558 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5766bf12-542b-458e-94a8-a564875de3b8" path="/var/lib/kubelet/pods/5766bf12-542b-458e-94a8-a564875de3b8/volumes" Jul 22 11:50:07.040972 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.040918 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-695df56d48-vbxtc" podUID="72a5c146-f3bb-40da-8bb5-b03704e76863" containerName="console" containerID="cri-o://1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9" gracePeriod=15 Jul 22 11:50:07.278554 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.278532 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:50:07.340775 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.340703 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-oauth-serving-cert\") pod \"72a5c146-f3bb-40da-8bb5-b03704e76863\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " Jul 22 11:50:07.340877 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.340793 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-serving-cert\") pod \"72a5c146-f3bb-40da-8bb5-b03704e76863\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " Jul 22 11:50:07.340877 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.340825 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-trusted-ca-bundle\") pod \"72a5c146-f3bb-40da-8bb5-b03704e76863\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " Jul 22 11:50:07.340877 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.340858 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-oauth-config\") pod \"72a5c146-f3bb-40da-8bb5-b03704e76863\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " Jul 22 11:50:07.340984 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.340907 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-service-ca\") pod \"72a5c146-f3bb-40da-8bb5-b03704e76863\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " Jul 22 11:50:07.340984 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.340934 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-console-config\") pod \"72a5c146-f3bb-40da-8bb5-b03704e76863\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " Jul 22 11:50:07.340984 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.340961 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv9bb\" (UniqueName: \"kubernetes.io/projected/72a5c146-f3bb-40da-8bb5-b03704e76863-kube-api-access-pv9bb\") pod \"72a5c146-f3bb-40da-8bb5-b03704e76863\" (UID: \"72a5c146-f3bb-40da-8bb5-b03704e76863\") " Jul 22 11:50:07.341252 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.341131 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "72a5c146-f3bb-40da-8bb5-b03704e76863" (UID: "72a5c146-f3bb-40da-8bb5-b03704e76863"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:50:07.341374 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.341263 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "72a5c146-f3bb-40da-8bb5-b03704e76863" (UID: "72a5c146-f3bb-40da-8bb5-b03704e76863"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:50:07.341374 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.341305 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-service-ca" (OuterVolumeSpecName: "service-ca") pod "72a5c146-f3bb-40da-8bb5-b03704e76863" (UID: "72a5c146-f3bb-40da-8bb5-b03704e76863"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:50:07.341374 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.341351 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-console-config" (OuterVolumeSpecName: "console-config") pod "72a5c146-f3bb-40da-8bb5-b03704e76863" (UID: "72a5c146-f3bb-40da-8bb5-b03704e76863"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 11:50:07.342790 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.342770 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "72a5c146-f3bb-40da-8bb5-b03704e76863" (UID: "72a5c146-f3bb-40da-8bb5-b03704e76863"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:50:07.342885 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.342832 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "72a5c146-f3bb-40da-8bb5-b03704e76863" (UID: "72a5c146-f3bb-40da-8bb5-b03704e76863"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:50:07.342944 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.342921 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a5c146-f3bb-40da-8bb5-b03704e76863-kube-api-access-pv9bb" (OuterVolumeSpecName: "kube-api-access-pv9bb") pod "72a5c146-f3bb-40da-8bb5-b03704e76863" (UID: "72a5c146-f3bb-40da-8bb5-b03704e76863"). InnerVolumeSpecName "kube-api-access-pv9bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:50:07.442445 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.442423 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-service-ca\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:50:07.442445 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.442442 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-console-config\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:50:07.442571 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.442451 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv9bb\" (UniqueName: \"kubernetes.io/projected/72a5c146-f3bb-40da-8bb5-b03704e76863-kube-api-access-pv9bb\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:50:07.442571 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.442459 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-oauth-serving-cert\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:50:07.442571 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.442468 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-serving-cert\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:50:07.442571 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.442476 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72a5c146-f3bb-40da-8bb5-b03704e76863-trusted-ca-bundle\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:50:07.442571 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:07.442485 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a5c146-f3bb-40da-8bb5-b03704e76863-console-oauth-config\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:50:08.089394 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:08.089356 2579 generic.go:356] "Generic (PLEG): container finished" podID="72a5c146-f3bb-40da-8bb5-b03704e76863" containerID="1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9" exitCode=0 Jul 22 11:50:08.089778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:08.089404 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695df56d48-vbxtc" Jul 22 11:50:08.089778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:08.089445 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695df56d48-vbxtc" event={"ID":"72a5c146-f3bb-40da-8bb5-b03704e76863","Type":"ContainerDied","Data":"1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9"} Jul 22 11:50:08.089778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:08.089520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695df56d48-vbxtc" event={"ID":"72a5c146-f3bb-40da-8bb5-b03704e76863","Type":"ContainerDied","Data":"07dd80a44031cd613202c33efb237befb39b4ce02617b9839b67f46f6c3d9ccf"} Jul 22 11:50:08.089778 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:08.089541 2579 scope.go:117] "RemoveContainer" containerID="1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9" Jul 22 11:50:08.098466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:08.098452 2579 scope.go:117] "RemoveContainer" containerID="1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9" Jul 22 11:50:08.098699 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:50:08.098683 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9\": container with ID starting with 1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9 not found: ID does not exist" containerID="1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9" Jul 22 11:50:08.098743 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:08.098708 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9"} err="failed to get container status \"1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9\": rpc error: code = NotFound desc = could not find container \"1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9\": container with ID starting with 1d6160a2ee8ca63d180a635d2cb90e64e53ac399109022aff75d671ba39575d9 not found: ID does not exist" Jul 22 11:50:08.111425 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:08.111404 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-695df56d48-vbxtc"] Jul 22 11:50:08.116793 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:08.116774 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-695df56d48-vbxtc"] Jul 22 11:50:08.205659 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:50:08.205638 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72a5c146-f3bb-40da-8bb5-b03704e76863" path="/var/lib/kubelet/pods/72a5c146-f3bb-40da-8bb5-b03704e76863/volumes" Jul 22 11:51:04.167292 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:51:04.167264 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:51:04.168059 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:51:04.168041 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:52:04.220782 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:52:04.220740 2579 scope.go:117] "RemoveContainer" containerID="26015d98439909863b57480a94894f723348523f8747114627c2a45cea2bdfab" Jul 22 11:52:04.228566 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:52:04.228550 2579 scope.go:117] "RemoveContainer" containerID="2bfd40dd77de445af2571b2f2c2b51a06f46fa4c4e2522788571a8b58d1d78fe" Jul 22 11:53:04.242080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:53:04.242044 2579 scope.go:117] "RemoveContainer" containerID="7d84b0b65ee3604aee49ddd23e7e625afa762bd6aa8a974475b98159c9fb351a" Jul 22 11:53:04.250763 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:53:04.250738 2579 scope.go:117] "RemoveContainer" containerID="7bc6d9bc68f6f47bed4d5016ad84bc10de7e56f26795f496a07d38e3f731037c" Jul 22 11:53:04.258770 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:53:04.258746 2579 scope.go:117] "RemoveContainer" containerID="f3d58564b5251ab17013eb17bd2370821b352b8f85af2d46d9cf776c9e871a8d" Jul 22 11:54:27.347522 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347490 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-25mzt"] Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347815 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="feast-apply" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347830 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="feast-apply" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347840 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="registry" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347846 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="registry" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347853 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a5c146-f3bb-40da-8bb5-b03704e76863" containerName="console" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347859 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a5c146-f3bb-40da-8bb5-b03704e76863" containerName="console" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347878 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="offline" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347905 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="offline" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347918 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="feast-init" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347925 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="feast-init" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347931 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="online" Jul 22 11:54:27.347970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347936 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="online" Jul 22 11:54:27.348537 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347988 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="registry" Jul 22 11:54:27.348537 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.347997 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="online" Jul 22 11:54:27.348537 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.348002 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a5c146-f3bb-40da-8bb5-b03704e76863" containerName="console" Jul 22 11:54:27.348537 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.348007 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5766bf12-542b-458e-94a8-a564875de3b8" containerName="offline" Jul 22 11:54:27.351109 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.351089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" Jul 22 11:54:27.353834 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.353815 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jul 22 11:54:27.353995 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.353978 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jul 22 11:54:27.355116 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.355099 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-h2d7c\"" Jul 22 11:54:27.369266 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.369245 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-25mzt"] Jul 22 11:54:27.465751 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.465721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f2c2\" (UniqueName: \"kubernetes.io/projected/36e0c665-84f6-45b7-b2d5-ed603e2732ab-kube-api-access-6f2c2\") pod \"redis-69bbd798b6-25mzt\" (UID: \"36e0c665-84f6-45b7-b2d5-ed603e2732ab\") " pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" Jul 22 11:54:27.566325 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.566297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f2c2\" (UniqueName: \"kubernetes.io/projected/36e0c665-84f6-45b7-b2d5-ed603e2732ab-kube-api-access-6f2c2\") pod \"redis-69bbd798b6-25mzt\" (UID: \"36e0c665-84f6-45b7-b2d5-ed603e2732ab\") " pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" Jul 22 11:54:27.575285 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.575257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f2c2\" (UniqueName: \"kubernetes.io/projected/36e0c665-84f6-45b7-b2d5-ed603e2732ab-kube-api-access-6f2c2\") pod \"redis-69bbd798b6-25mzt\" (UID: \"36e0c665-84f6-45b7-b2d5-ed603e2732ab\") " pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" Jul 22 11:54:27.674612 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.674581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" Jul 22 11:54:27.802097 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.802072 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-25mzt"] Jul 22 11:54:27.804640 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:54:27.804612 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36e0c665_84f6_45b7_b2d5_ed603e2732ab.slice/crio-34a9641d75a8b526c195ec0c5d92fe888d025ffa883082be9f72d894feef53cd WatchSource:0}: Error finding container 34a9641d75a8b526c195ec0c5d92fe888d025ffa883082be9f72d894feef53cd: Status 404 returned error can't find the container with id 34a9641d75a8b526c195ec0c5d92fe888d025ffa883082be9f72d894feef53cd Jul 22 11:54:27.806365 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:27.806347 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 11:54:28.186701 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:28.186668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" event={"ID":"36e0c665-84f6-45b7-b2d5-ed603e2732ab","Type":"ContainerStarted","Data":"34a9641d75a8b526c195ec0c5d92fe888d025ffa883082be9f72d894feef53cd"} Jul 22 11:54:30.197779 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:30.197746 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" event={"ID":"36e0c665-84f6-45b7-b2d5-ed603e2732ab","Type":"ContainerStarted","Data":"95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2"} Jul 22 11:54:30.221906 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:30.221858 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" podStartSLOduration=1.575696852 podStartE2EDuration="3.221841715s" totalImagesPullingTime="1.646144863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:54:30.220558321 +0000 UTC m=+1106.519533278" watchObservedRunningTime="2026-07-22 11:54:30.221841715 +0000 UTC m=+1106.520816670" Jul 22 11:54:32.916329 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:32.916295 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nvgh2"] Jul 22 11:54:32.919478 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:32.919457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" Jul 22 11:54:32.922737 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:32.922712 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jul 22 11:54:32.929605 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:32.929570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nvgh2"] Jul 22 11:54:33.013664 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:33.013630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-postgresdata\") pod \"postgres-864846d4f5-nvgh2\" (UID: \"f2cfa467-b4c8-4a43-a6d7-4ddb53840612\") " pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" Jul 22 11:54:33.013808 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:33.013667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4bsx\" (UniqueName: \"kubernetes.io/projected/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-kube-api-access-j4bsx\") pod \"postgres-864846d4f5-nvgh2\" (UID: \"f2cfa467-b4c8-4a43-a6d7-4ddb53840612\") " pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" Jul 22 11:54:33.114890 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:33.114856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-postgresdata\") pod \"postgres-864846d4f5-nvgh2\" (UID: \"f2cfa467-b4c8-4a43-a6d7-4ddb53840612\") " pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" Jul 22 11:54:33.115033 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:33.114890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4bsx\" (UniqueName: \"kubernetes.io/projected/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-kube-api-access-j4bsx\") pod \"postgres-864846d4f5-nvgh2\" (UID: \"f2cfa467-b4c8-4a43-a6d7-4ddb53840612\") " pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" Jul 22 11:54:33.115238 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:33.115217 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-postgresdata\") pod \"postgres-864846d4f5-nvgh2\" (UID: \"f2cfa467-b4c8-4a43-a6d7-4ddb53840612\") " pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" Jul 22 11:54:33.123797 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:33.123777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4bsx\" (UniqueName: \"kubernetes.io/projected/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-kube-api-access-j4bsx\") pod \"postgres-864846d4f5-nvgh2\" (UID: \"f2cfa467-b4c8-4a43-a6d7-4ddb53840612\") " pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" Jul 22 11:54:33.228839 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:33.228767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" Jul 22 11:54:33.362544 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:33.362520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nvgh2"] Jul 22 11:54:33.364480 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:54:33.364453 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2cfa467_b4c8_4a43_a6d7_4ddb53840612.slice/crio-1d427719ef7d7a93f48500a403c4f0bd5ae603d30e32b3ce95f3aca018f17830 WatchSource:0}: Error finding container 1d427719ef7d7a93f48500a403c4f0bd5ae603d30e32b3ce95f3aca018f17830: Status 404 returned error can't find the container with id 1d427719ef7d7a93f48500a403c4f0bd5ae603d30e32b3ce95f3aca018f17830 Jul 22 11:54:34.215647 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:34.215609 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" event={"ID":"f2cfa467-b4c8-4a43-a6d7-4ddb53840612","Type":"ContainerStarted","Data":"296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1"} Jul 22 11:54:34.215647 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:34.215650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" event={"ID":"f2cfa467-b4c8-4a43-a6d7-4ddb53840612","Type":"ContainerStarted","Data":"1d427719ef7d7a93f48500a403c4f0bd5ae603d30e32b3ce95f3aca018f17830"} Jul 22 11:54:34.231615 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:34.231571 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" podStartSLOduration=2.231557599 podStartE2EDuration="2.231557599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:54:34.23101012 +0000 UTC m=+1110.529985080" watchObservedRunningTime="2026-07-22 11:54:34.231557599 +0000 UTC m=+1110.530532555" Jul 22 11:54:38.825812 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.825777 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22"] Jul 22 11:54:38.829332 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.829312 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.834387 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.834363 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jul 22 11:54:38.834837 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.834814 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jul 22 11:54:38.834925 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.834882 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-nkk5r\"" Jul 22 11:54:38.835170 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.835153 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jul 22 11:54:38.859306 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.859282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-registry-tls\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.859440 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.859341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4z2\" (UniqueName: \"kubernetes.io/projected/7b2f4790-8e41-43e8-acb0-06b9b53dab65-kube-api-access-fw4z2\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.859440 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.859372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2f4790-8e41-43e8-acb0-06b9b53dab65-feast-data\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.859440 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.859404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-online-tls\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.867058 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.867037 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22"] Jul 22 11:54:38.959992 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.959964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4z2\" (UniqueName: \"kubernetes.io/projected/7b2f4790-8e41-43e8-acb0-06b9b53dab65-kube-api-access-fw4z2\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.960151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.960009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2f4790-8e41-43e8-acb0-06b9b53dab65-feast-data\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.960256 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.960240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-online-tls\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.960341 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.960325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-registry-tls\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.960453 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:54:38.960439 2579 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Jul 22 11:54:38.960513 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:54:38.960496 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-registry-tls podName:7b2f4790-8e41-43e8-acb0-06b9b53dab65 nodeName:}" failed. No retries permitted until 2026-07-22 11:54:39.460481967 +0000 UTC m=+1115.759456903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-registry-tls") pod "feast-credit-scoring-5bc485d5c-nrg22" (UID: "7b2f4790-8e41-43e8-acb0-06b9b53dab65") : secret "feast-credit-scoring-registry-tls" not found Jul 22 11:54:38.960570 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.960505 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2f4790-8e41-43e8-acb0-06b9b53dab65-feast-data\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.962567 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.962548 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-online-tls\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:38.976876 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:38.976856 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4z2\" (UniqueName: \"kubernetes.io/projected/7b2f4790-8e41-43e8-acb0-06b9b53dab65-kube-api-access-fw4z2\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:39.465367 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:39.465331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-registry-tls\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:39.467782 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:39.467760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-registry-tls\") pod \"feast-credit-scoring-5bc485d5c-nrg22\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:39.738642 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:39.738547 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:39.880864 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:39.880839 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22"] Jul 22 11:54:39.882646 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:54:39.882610 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2f4790_8e41_43e8_acb0_06b9b53dab65.slice/crio-7b1dc2405ed298453cff912a02c79d6751d9db87f43efd816def69757347b04e WatchSource:0}: Error finding container 7b1dc2405ed298453cff912a02c79d6751d9db87f43efd816def69757347b04e: Status 404 returned error can't find the container with id 7b1dc2405ed298453cff912a02c79d6751d9db87f43efd816def69757347b04e Jul 22 11:54:40.242751 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:40.242708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" event={"ID":"7b2f4790-8e41-43e8-acb0-06b9b53dab65","Type":"ContainerStarted","Data":"7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7"} Jul 22 11:54:40.242947 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:40.242770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" event={"ID":"7b2f4790-8e41-43e8-acb0-06b9b53dab65","Type":"ContainerStarted","Data":"7b1dc2405ed298453cff912a02c79d6751d9db87f43efd816def69757347b04e"} Jul 22 11:54:41.248054 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:41.248015 2579 generic.go:356] "Generic (PLEG): container finished" podID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerID="7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7" exitCode=0 Jul 22 11:54:41.248452 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:41.248095 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" event={"ID":"7b2f4790-8e41-43e8-acb0-06b9b53dab65","Type":"ContainerDied","Data":"7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7"} Jul 22 11:54:42.254159 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:42.254114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" event={"ID":"7b2f4790-8e41-43e8-acb0-06b9b53dab65","Type":"ContainerStarted","Data":"2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f"} Jul 22 11:54:47.921049 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:54:47.921018 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2f4790_8e41_43e8_acb0_06b9b53dab65.slice/crio-conmon-2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f.scope\": RecentStats: unable to find data in memory cache]" Jul 22 11:54:48.281357 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:48.281278 2579 generic.go:356] "Generic (PLEG): container finished" podID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerID="2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f" exitCode=0 Jul 22 11:54:48.281357 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:48.281331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" event={"ID":"7b2f4790-8e41-43e8-acb0-06b9b53dab65","Type":"ContainerDied","Data":"2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f"} Jul 22 11:54:49.288999 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:49.288962 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" event={"ID":"7b2f4790-8e41-43e8-acb0-06b9b53dab65","Type":"ContainerStarted","Data":"0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3"} Jul 22 11:54:49.289520 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:49.289018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" event={"ID":"7b2f4790-8e41-43e8-acb0-06b9b53dab65","Type":"ContainerStarted","Data":"0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61"} Jul 22 11:54:49.313113 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:49.313043 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podStartSLOduration=11.313025677 podStartE2EDuration="11.313025677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:54:49.311776064 +0000 UTC m=+1125.610751019" watchObservedRunningTime="2026-07-22 11:54:49.313025677 +0000 UTC m=+1125.612000635" Jul 22 11:54:49.739287 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:49.739241 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:49.739466 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:49.739326 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:51.739394 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:51.739353 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:51.739798 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:51.739434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:51.740971 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:51.740941 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Jul 22 11:54:51.741107 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:51.740987 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jul 22 11:54:54.739525 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:54.739481 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Jul 22 11:54:54.744960 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:54.744932 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:54.748043 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:54.748015 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:54:57.739698 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:54:57.739660 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Jul 22 11:55:00.740226 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:00.740176 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:55:00.740924 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:00.740906 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:55:04.755009 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.754972 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr"] Jul 22 11:55:04.757842 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.757821 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.761946 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.761921 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jul 22 11:55:04.762085 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.761976 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-vkmp9\"" Jul 22 11:55:04.762374 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.762353 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jul 22 11:55:04.762465 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.762357 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jul 22 11:55:04.775310 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.775282 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr"] Jul 22 11:55:04.884361 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.884329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/163ee316-d300-4a21-964b-7ef70815f18e-feast-data\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.884548 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.884389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-online-tls\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.884548 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.884462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpg7k\" (UniqueName: \"kubernetes.io/projected/163ee316-d300-4a21-964b-7ef70815f18e-kube-api-access-wpg7k\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.884548 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.884512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-registry-tls\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.984957 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.984920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-online-tls\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.985129 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.984976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpg7k\" (UniqueName: \"kubernetes.io/projected/163ee316-d300-4a21-964b-7ef70815f18e-kube-api-access-wpg7k\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.985129 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.985010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-registry-tls\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.985129 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.985029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/163ee316-d300-4a21-964b-7ef70815f18e-feast-data\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.985455 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.985421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/163ee316-d300-4a21-964b-7ef70815f18e-feast-data\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.987557 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.987532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-registry-tls\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.987663 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.987532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-online-tls\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:04.995752 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:04.995724 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpg7k\" (UniqueName: \"kubernetes.io/projected/163ee316-d300-4a21-964b-7ef70815f18e-kube-api-access-wpg7k\") pod \"feast-driver-ranking-5cf9786f5d-xtvmr\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:05.067881 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:05.067813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:05.203811 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:05.203787 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr"] Jul 22 11:55:05.205603 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:55:05.205575 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163ee316_d300_4a21_964b_7ef70815f18e.slice/crio-087240c0fd14c1f10ee00af10977223bdf7a2ac55a711ad3467df4050176d927 WatchSource:0}: Error finding container 087240c0fd14c1f10ee00af10977223bdf7a2ac55a711ad3467df4050176d927: Status 404 returned error can't find the container with id 087240c0fd14c1f10ee00af10977223bdf7a2ac55a711ad3467df4050176d927 Jul 22 11:55:05.363172 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:05.363083 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" event={"ID":"163ee316-d300-4a21-964b-7ef70815f18e","Type":"ContainerStarted","Data":"c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b"} Jul 22 11:55:05.363172 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:05.363141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" event={"ID":"163ee316-d300-4a21-964b-7ef70815f18e","Type":"ContainerStarted","Data":"087240c0fd14c1f10ee00af10977223bdf7a2ac55a711ad3467df4050176d927"} Jul 22 11:55:10.389621 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:10.389587 2579 generic.go:356] "Generic (PLEG): container finished" podID="163ee316-d300-4a21-964b-7ef70815f18e" containerID="c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b" exitCode=0 Jul 22 11:55:10.390047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:10.389650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" event={"ID":"163ee316-d300-4a21-964b-7ef70815f18e","Type":"ContainerDied","Data":"c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b"} Jul 22 11:55:11.395308 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:11.395277 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" event={"ID":"163ee316-d300-4a21-964b-7ef70815f18e","Type":"ContainerStarted","Data":"a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018"} Jul 22 11:55:17.423814 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:17.423777 2579 generic.go:356] "Generic (PLEG): container finished" podID="163ee316-d300-4a21-964b-7ef70815f18e" containerID="a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018" exitCode=0 Jul 22 11:55:17.424294 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:17.423818 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" event={"ID":"163ee316-d300-4a21-964b-7ef70815f18e","Type":"ContainerDied","Data":"a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018"} Jul 22 11:55:18.432479 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:18.432447 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" event={"ID":"163ee316-d300-4a21-964b-7ef70815f18e","Type":"ContainerStarted","Data":"3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded"} Jul 22 11:55:18.432942 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:18.432508 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" event={"ID":"163ee316-d300-4a21-964b-7ef70815f18e","Type":"ContainerStarted","Data":"75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5"} Jul 22 11:55:18.456713 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:18.456660 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podStartSLOduration=14.456644265 podStartE2EDuration="14.456644265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:55:18.454364709 +0000 UTC m=+1154.753339685" watchObservedRunningTime="2026-07-22 11:55:18.456644265 +0000 UTC m=+1154.755619221" Jul 22 11:55:20.068450 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:20.068410 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:20.068908 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:20.068484 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:20.069980 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:20.069951 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Jul 22 11:55:20.070114 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:20.069996 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jul 22 11:55:23.069983 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:23.069926 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Jul 22 11:55:23.420522 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:23.420489 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:23.420922 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:23.420899 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:23.425029 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:23.425003 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:25.069175 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:25.069134 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:26.068404 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:26.068357 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Jul 22 11:55:29.068821 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:29.068713 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Jul 22 11:55:32.069131 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:32.069096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:55:32.069917 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:55:32.069898 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:56:04.199576 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:04.199544 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:56:04.200322 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:04.200301 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 11:56:49.852412 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:49.852371 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc"] Jul 22 11:56:49.852934 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:49.852615 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" podUID="ebc4b953-2a31-4f79-8157-ba40eb91ba5f" containerName="manager" containerID="cri-o://efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4" gracePeriod=10 Jul 22 11:56:50.124521 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.124493 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" Jul 22 11:56:50.205929 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.205900 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hptvw\" (UniqueName: \"kubernetes.io/projected/ebc4b953-2a31-4f79-8157-ba40eb91ba5f-kube-api-access-hptvw\") pod \"ebc4b953-2a31-4f79-8157-ba40eb91ba5f\" (UID: \"ebc4b953-2a31-4f79-8157-ba40eb91ba5f\") " Jul 22 11:56:50.208156 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.208120 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc4b953-2a31-4f79-8157-ba40eb91ba5f-kube-api-access-hptvw" (OuterVolumeSpecName: "kube-api-access-hptvw") pod "ebc4b953-2a31-4f79-8157-ba40eb91ba5f" (UID: "ebc4b953-2a31-4f79-8157-ba40eb91ba5f"). InnerVolumeSpecName "kube-api-access-hptvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:56:50.221569 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.221537 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22"] Jul 22 11:56:50.221788 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.221754 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="registry" containerID="cri-o://0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61" gracePeriod=30 Jul 22 11:56:50.221886 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.221801 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="online" containerID="cri-o://0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3" gracePeriod=30 Jul 22 11:56:50.236970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.236940 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr"] Jul 22 11:56:50.237177 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.237152 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="registry" containerID="cri-o://75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5" gracePeriod=30 Jul 22 11:56:50.237259 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.237183 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="online" containerID="cri-o://3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded" gracePeriod=30 Jul 22 11:56:50.307394 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.307360 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hptvw\" (UniqueName: \"kubernetes.io/projected/ebc4b953-2a31-4f79-8157-ba40eb91ba5f-kube-api-access-hptvw\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:56:50.904558 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.904512 2579 generic.go:356] "Generic (PLEG): container finished" podID="ebc4b953-2a31-4f79-8157-ba40eb91ba5f" containerID="efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4" exitCode=0 Jul 22 11:56:50.904558 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.904554 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" Jul 22 11:56:50.905077 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.904601 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" event={"ID":"ebc4b953-2a31-4f79-8157-ba40eb91ba5f","Type":"ContainerDied","Data":"efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4"} Jul 22 11:56:50.905077 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.904656 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc" event={"ID":"ebc4b953-2a31-4f79-8157-ba40eb91ba5f","Type":"ContainerDied","Data":"83349ca3827301ee232e3852ada3aa6fe141fb9887118ed5eb87ab25afdcea11"} Jul 22 11:56:50.905077 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.904683 2579 scope.go:117] "RemoveContainer" containerID="efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4" Jul 22 11:56:50.915003 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.914981 2579 scope.go:117] "RemoveContainer" containerID="efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4" Jul 22 11:56:50.915332 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:56:50.915305 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4\": container with ID starting with efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4 not found: ID does not exist" containerID="efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4" Jul 22 11:56:50.915415 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.915345 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4"} err="failed to get container status \"efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4\": rpc error: code = NotFound desc = could not find container \"efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4\": container with ID starting with efee7874dfcda4c28e79983a685868f31f6cc2649bb967851151808f413141b4 not found: ID does not exist" Jul 22 11:56:50.927290 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.927266 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc"] Jul 22 11:56:50.930761 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:50.930737 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7c86c97887-tmjzc"] Jul 22 11:56:51.911686 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:51.911652 2579 generic.go:356] "Generic (PLEG): container finished" podID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerID="0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3" exitCode=0 Jul 22 11:56:51.912151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:51.911724 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" event={"ID":"7b2f4790-8e41-43e8-acb0-06b9b53dab65","Type":"ContainerDied","Data":"0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3"} Jul 22 11:56:51.914731 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:51.914711 2579 generic.go:356] "Generic (PLEG): container finished" podID="163ee316-d300-4a21-964b-7ef70815f18e" containerID="3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded" exitCode=0 Jul 22 11:56:51.914845 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:51.914786 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" event={"ID":"163ee316-d300-4a21-964b-7ef70815f18e","Type":"ContainerDied","Data":"3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded"} Jul 22 11:56:52.209524 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.209445 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc4b953-2a31-4f79-8157-ba40eb91ba5f" path="/var/lib/kubelet/pods/ebc4b953-2a31-4f79-8157-ba40eb91ba5f/volumes" Jul 22 11:56:52.271922 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.271888 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nvgh2"] Jul 22 11:56:52.272089 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.272066 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" podUID="f2cfa467-b4c8-4a43-a6d7-4ddb53840612" containerName="postgres" containerID="cri-o://296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1" gracePeriod=30 Jul 22 11:56:52.277984 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.277956 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-25mzt"] Jul 22 11:56:52.278456 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.278426 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" podUID="36e0c665-84f6-45b7-b2d5-ed603e2732ab" containerName="redis" containerID="cri-o://95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2" gracePeriod=30 Jul 22 11:56:52.550277 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.550251 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" Jul 22 11:56:52.552954 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.552928 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" Jul 22 11:56:52.629870 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.629832 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f2c2\" (UniqueName: \"kubernetes.io/projected/36e0c665-84f6-45b7-b2d5-ed603e2732ab-kube-api-access-6f2c2\") pod \"36e0c665-84f6-45b7-b2d5-ed603e2732ab\" (UID: \"36e0c665-84f6-45b7-b2d5-ed603e2732ab\") " Jul 22 11:56:52.630043 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.629878 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-postgresdata\") pod \"f2cfa467-b4c8-4a43-a6d7-4ddb53840612\" (UID: \"f2cfa467-b4c8-4a43-a6d7-4ddb53840612\") " Jul 22 11:56:52.630043 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.629937 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4bsx\" (UniqueName: \"kubernetes.io/projected/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-kube-api-access-j4bsx\") pod \"f2cfa467-b4c8-4a43-a6d7-4ddb53840612\" (UID: \"f2cfa467-b4c8-4a43-a6d7-4ddb53840612\") " Jul 22 11:56:52.632048 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.632018 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-kube-api-access-j4bsx" (OuterVolumeSpecName: "kube-api-access-j4bsx") pod "f2cfa467-b4c8-4a43-a6d7-4ddb53840612" (UID: "f2cfa467-b4c8-4a43-a6d7-4ddb53840612"). InnerVolumeSpecName "kube-api-access-j4bsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:56:52.632182 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.632093 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e0c665-84f6-45b7-b2d5-ed603e2732ab-kube-api-access-6f2c2" (OuterVolumeSpecName: "kube-api-access-6f2c2") pod "36e0c665-84f6-45b7-b2d5-ed603e2732ab" (UID: "36e0c665-84f6-45b7-b2d5-ed603e2732ab"). InnerVolumeSpecName "kube-api-access-6f2c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:56:52.660512 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.660485 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "f2cfa467-b4c8-4a43-a6d7-4ddb53840612" (UID: "f2cfa467-b4c8-4a43-a6d7-4ddb53840612"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:56:52.730560 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.730495 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f2c2\" (UniqueName: \"kubernetes.io/projected/36e0c665-84f6-45b7-b2d5-ed603e2732ab-kube-api-access-6f2c2\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:56:52.730560 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.730519 2579 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-postgresdata\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:56:52.730560 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.730528 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4bsx\" (UniqueName: \"kubernetes.io/projected/f2cfa467-b4c8-4a43-a6d7-4ddb53840612-kube-api-access-j4bsx\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:56:52.920580 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.920543 2579 generic.go:356] "Generic (PLEG): container finished" podID="f2cfa467-b4c8-4a43-a6d7-4ddb53840612" containerID="296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1" exitCode=0 Jul 22 11:56:52.920580 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.920583 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" Jul 22 11:56:52.921105 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.920631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" event={"ID":"f2cfa467-b4c8-4a43-a6d7-4ddb53840612","Type":"ContainerDied","Data":"296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1"} Jul 22 11:56:52.921105 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.920671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nvgh2" event={"ID":"f2cfa467-b4c8-4a43-a6d7-4ddb53840612","Type":"ContainerDied","Data":"1d427719ef7d7a93f48500a403c4f0bd5ae603d30e32b3ce95f3aca018f17830"} Jul 22 11:56:52.921105 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.920690 2579 scope.go:117] "RemoveContainer" containerID="296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1" Jul 22 11:56:52.921929 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.921908 2579 generic.go:356] "Generic (PLEG): container finished" podID="36e0c665-84f6-45b7-b2d5-ed603e2732ab" containerID="95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2" exitCode=0 Jul 22 11:56:52.922038 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.921943 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" event={"ID":"36e0c665-84f6-45b7-b2d5-ed603e2732ab","Type":"ContainerDied","Data":"95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2"} Jul 22 11:56:52.922038 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.921960 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" Jul 22 11:56:52.922038 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.921985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-25mzt" event={"ID":"36e0c665-84f6-45b7-b2d5-ed603e2732ab","Type":"ContainerDied","Data":"34a9641d75a8b526c195ec0c5d92fe888d025ffa883082be9f72d894feef53cd"} Jul 22 11:56:52.930806 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.930784 2579 scope.go:117] "RemoveContainer" containerID="296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1" Jul 22 11:56:52.931105 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:56:52.931078 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1\": container with ID starting with 296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1 not found: ID does not exist" containerID="296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1" Jul 22 11:56:52.931227 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.931113 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1"} err="failed to get container status \"296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1\": rpc error: code = NotFound desc = could not find container \"296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1\": container with ID starting with 296090865b8757ab8490f1b519675e17caf45a0c5bcbcd7ba08d4a8d746e88d1 not found: ID does not exist" Jul 22 11:56:52.931227 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.931135 2579 scope.go:117] "RemoveContainer" containerID="95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2" Jul 22 11:56:52.938731 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.938712 2579 scope.go:117] "RemoveContainer" containerID="95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2" Jul 22 11:56:52.938984 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:56:52.938963 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2\": container with ID starting with 95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2 not found: ID does not exist" containerID="95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2" Jul 22 11:56:52.939107 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.938989 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2"} err="failed to get container status \"95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2\": rpc error: code = NotFound desc = could not find container \"95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2\": container with ID starting with 95805af4e4ff5bb29cb8eefae316752d46467f03c20fd7b9355e9987bd0671b2 not found: ID does not exist" Jul 22 11:56:52.943126 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.943105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nvgh2"] Jul 22 11:56:52.949433 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.949409 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nvgh2"] Jul 22 11:56:52.959866 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.959846 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-25mzt"] Jul 22 11:56:52.963644 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:52.963622 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-25mzt"] Jul 22 11:56:53.421792 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:53.421748 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jul 22 11:56:54.206214 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:54.206172 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36e0c665-84f6-45b7-b2d5-ed603e2732ab" path="/var/lib/kubelet/pods/36e0c665-84f6-45b7-b2d5-ed603e2732ab/volumes" Jul 22 11:56:54.206623 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:54.206572 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cfa467-b4c8-4a43-a6d7-4ddb53840612" path="/var/lib/kubelet/pods/f2cfa467-b4c8-4a43-a6d7-4ddb53840612/volumes" Jul 22 11:56:54.746458 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:56:54.746412 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jul 22 11:57:03.422031 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:03.421981 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jul 22 11:57:04.745736 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:04.745687 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jul 22 11:57:13.421714 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:13.421663 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jul 22 11:57:13.422360 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:13.421808 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:57:14.746038 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:14.745988 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jul 22 11:57:14.746523 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:14.746126 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:57:20.740937 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.740899 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Jul 22 11:57:20.893970 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.893936 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:57:20.909668 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.909641 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:57:20.970015 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.969976 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-registry-tls\") pod \"163ee316-d300-4a21-964b-7ef70815f18e\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " Jul 22 11:57:20.970251 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.970054 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-online-tls\") pod \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " Jul 22 11:57:20.970251 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.970077 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-online-tls\") pod \"163ee316-d300-4a21-964b-7ef70815f18e\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " Jul 22 11:57:20.970251 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.970154 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-registry-tls\") pod \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " Jul 22 11:57:20.970251 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.970182 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw4z2\" (UniqueName: \"kubernetes.io/projected/7b2f4790-8e41-43e8-acb0-06b9b53dab65-kube-api-access-fw4z2\") pod \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " Jul 22 11:57:20.970470 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.970257 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpg7k\" (UniqueName: \"kubernetes.io/projected/163ee316-d300-4a21-964b-7ef70815f18e-kube-api-access-wpg7k\") pod \"163ee316-d300-4a21-964b-7ef70815f18e\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " Jul 22 11:57:20.970470 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.970285 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2f4790-8e41-43e8-acb0-06b9b53dab65-feast-data\") pod \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\" (UID: \"7b2f4790-8e41-43e8-acb0-06b9b53dab65\") " Jul 22 11:57:20.970470 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.970314 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/163ee316-d300-4a21-964b-7ef70815f18e-feast-data\") pod \"163ee316-d300-4a21-964b-7ef70815f18e\" (UID: \"163ee316-d300-4a21-964b-7ef70815f18e\") " Jul 22 11:57:20.971045 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.971020 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163ee316-d300-4a21-964b-7ef70815f18e-feast-data" (OuterVolumeSpecName: "feast-data") pod "163ee316-d300-4a21-964b-7ef70815f18e" (UID: "163ee316-d300-4a21-964b-7ef70815f18e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:57:20.972624 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.972582 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7b2f4790-8e41-43e8-acb0-06b9b53dab65" (UID: "7b2f4790-8e41-43e8-acb0-06b9b53dab65"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:57:20.972754 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.972732 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-online-tls" (OuterVolumeSpecName: "online-tls") pod "163ee316-d300-4a21-964b-7ef70815f18e" (UID: "163ee316-d300-4a21-964b-7ef70815f18e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:57:20.972854 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.972820 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2f4790-8e41-43e8-acb0-06b9b53dab65-kube-api-access-fw4z2" (OuterVolumeSpecName: "kube-api-access-fw4z2") pod "7b2f4790-8e41-43e8-acb0-06b9b53dab65" (UID: "7b2f4790-8e41-43e8-acb0-06b9b53dab65"). InnerVolumeSpecName "kube-api-access-fw4z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:57:20.972854 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.972841 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "163ee316-d300-4a21-964b-7ef70815f18e" (UID: "163ee316-d300-4a21-964b-7ef70815f18e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:57:20.973002 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.972840 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-online-tls" (OuterVolumeSpecName: "online-tls") pod "7b2f4790-8e41-43e8-acb0-06b9b53dab65" (UID: "7b2f4790-8e41-43e8-acb0-06b9b53dab65"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 11:57:20.973125 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.973109 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163ee316-d300-4a21-964b-7ef70815f18e-kube-api-access-wpg7k" (OuterVolumeSpecName: "kube-api-access-wpg7k") pod "163ee316-d300-4a21-964b-7ef70815f18e" (UID: "163ee316-d300-4a21-964b-7ef70815f18e"). InnerVolumeSpecName "kube-api-access-wpg7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 11:57:20.980046 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:20.980012 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2f4790-8e41-43e8-acb0-06b9b53dab65-feast-data" (OuterVolumeSpecName: "feast-data") pod "7b2f4790-8e41-43e8-acb0-06b9b53dab65" (UID: "7b2f4790-8e41-43e8-acb0-06b9b53dab65"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 11:57:21.049540 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.049447 2579 generic.go:356] "Generic (PLEG): container finished" podID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerID="0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61" exitCode=137 Jul 22 11:57:21.049540 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.049502 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" Jul 22 11:57:21.049787 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.049534 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" event={"ID":"7b2f4790-8e41-43e8-acb0-06b9b53dab65","Type":"ContainerDied","Data":"0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61"} Jul 22 11:57:21.049787 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.049607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22" event={"ID":"7b2f4790-8e41-43e8-acb0-06b9b53dab65","Type":"ContainerDied","Data":"7b1dc2405ed298453cff912a02c79d6751d9db87f43efd816def69757347b04e"} Jul 22 11:57:21.049787 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.049629 2579 scope.go:117] "RemoveContainer" containerID="0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3" Jul 22 11:57:21.051948 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.051924 2579 generic.go:356] "Generic (PLEG): container finished" podID="163ee316-d300-4a21-964b-7ef70815f18e" containerID="75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5" exitCode=137 Jul 22 11:57:21.052088 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.051976 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" Jul 22 11:57:21.052088 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.051983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" event={"ID":"163ee316-d300-4a21-964b-7ef70815f18e","Type":"ContainerDied","Data":"75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5"} Jul 22 11:57:21.052088 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.052027 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr" event={"ID":"163ee316-d300-4a21-964b-7ef70815f18e","Type":"ContainerDied","Data":"087240c0fd14c1f10ee00af10977223bdf7a2ac55a711ad3467df4050176d927"} Jul 22 11:57:21.060575 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.060549 2579 scope.go:117] "RemoveContainer" containerID="0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61" Jul 22 11:57:21.068686 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.068664 2579 scope.go:117] "RemoveContainer" containerID="2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f" Jul 22 11:57:21.070981 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.070959 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:57:21.071080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.070987 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw4z2\" (UniqueName: \"kubernetes.io/projected/7b2f4790-8e41-43e8-acb0-06b9b53dab65-kube-api-access-fw4z2\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:57:21.071080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.071003 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpg7k\" (UniqueName: \"kubernetes.io/projected/163ee316-d300-4a21-964b-7ef70815f18e-kube-api-access-wpg7k\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:57:21.071080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.071016 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2f4790-8e41-43e8-acb0-06b9b53dab65-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:57:21.071080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.071029 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/163ee316-d300-4a21-964b-7ef70815f18e-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:57:21.071080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.071041 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:57:21.071080 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.071080 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7b2f4790-8e41-43e8-acb0-06b9b53dab65-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:57:21.071365 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.071093 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/163ee316-d300-4a21-964b-7ef70815f18e-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 11:57:21.074697 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.074667 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22"] Jul 22 11:57:21.077456 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.077433 2579 scope.go:117] "RemoveContainer" containerID="7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7" Jul 22 11:57:21.080407 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.080382 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5bc485d5c-nrg22"] Jul 22 11:57:21.085919 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.085897 2579 scope.go:117] "RemoveContainer" containerID="0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3" Jul 22 11:57:21.086281 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:57:21.086259 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3\": container with ID starting with 0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3 not found: ID does not exist" containerID="0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3" Jul 22 11:57:21.086331 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.086293 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3"} err="failed to get container status \"0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3\": rpc error: code = NotFound desc = could not find container \"0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3\": container with ID starting with 0a9dc75b82f38d8c8165407f2637197fa4071ac9614a10f46a8427f9b96890f3 not found: ID does not exist" Jul 22 11:57:21.086331 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.086315 2579 scope.go:117] "RemoveContainer" containerID="0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61" Jul 22 11:57:21.086572 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:57:21.086549 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61\": container with ID starting with 0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61 not found: ID does not exist" containerID="0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61" Jul 22 11:57:21.086674 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.086576 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61"} err="failed to get container status \"0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61\": rpc error: code = NotFound desc = could not find container \"0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61\": container with ID starting with 0a6499db7ec3e97ef448551874c3de66baeaba4580d8563fbe3db0d53bc87f61 not found: ID does not exist" Jul 22 11:57:21.086674 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.086597 2579 scope.go:117] "RemoveContainer" containerID="2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f" Jul 22 11:57:21.086874 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:57:21.086847 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f\": container with ID starting with 2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f not found: ID does not exist" containerID="2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f" Jul 22 11:57:21.086921 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.086878 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f"} err="failed to get container status \"2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f\": rpc error: code = NotFound desc = could not find container \"2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f\": container with ID starting with 2cc8abfddd1712f3c6e87460c2ea56baa821e0a453505e453325195b8574e46f not found: ID does not exist" Jul 22 11:57:21.086921 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.086892 2579 scope.go:117] "RemoveContainer" containerID="7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7" Jul 22 11:57:21.087136 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:57:21.087119 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7\": container with ID starting with 7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7 not found: ID does not exist" containerID="7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7" Jul 22 11:57:21.087182 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.087142 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7"} err="failed to get container status \"7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7\": rpc error: code = NotFound desc = could not find container \"7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7\": container with ID starting with 7cbfaddd5a9f92f302222e159d33b331c955ac9f3bf3c450f90d575ac4647ca7 not found: ID does not exist" Jul 22 11:57:21.087182 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.087156 2579 scope.go:117] "RemoveContainer" containerID="3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded" Jul 22 11:57:21.093914 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.093890 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr"] Jul 22 11:57:21.096564 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.096248 2579 scope.go:117] "RemoveContainer" containerID="75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5" Jul 22 11:57:21.097763 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.097743 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5cf9786f5d-xtvmr"] Jul 22 11:57:21.104642 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.104625 2579 scope.go:117] "RemoveContainer" containerID="a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018" Jul 22 11:57:21.112444 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.112425 2579 scope.go:117] "RemoveContainer" containerID="c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b" Jul 22 11:57:21.120177 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.120157 2579 scope.go:117] "RemoveContainer" containerID="3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded" Jul 22 11:57:21.120500 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:57:21.120479 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded\": container with ID starting with 3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded not found: ID does not exist" containerID="3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded" Jul 22 11:57:21.120552 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.120508 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded"} err="failed to get container status \"3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded\": rpc error: code = NotFound desc = could not find container \"3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded\": container with ID starting with 3875f65cb5fc7390fc5fed8713351bdff38df23be8dac106b6a4e9856b783ded not found: ID does not exist" Jul 22 11:57:21.120552 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.120530 2579 scope.go:117] "RemoveContainer" containerID="75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5" Jul 22 11:57:21.120794 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:57:21.120773 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5\": container with ID starting with 75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5 not found: ID does not exist" containerID="75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5" Jul 22 11:57:21.120839 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.120801 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5"} err="failed to get container status \"75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5\": rpc error: code = NotFound desc = could not find container \"75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5\": container with ID starting with 75fbbd43e75dca7bae86f61745c1998a722c315d0b15ba247568927c542507e5 not found: ID does not exist" Jul 22 11:57:21.120839 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.120818 2579 scope.go:117] "RemoveContainer" containerID="a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018" Jul 22 11:57:21.121086 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:57:21.121057 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018\": container with ID starting with a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018 not found: ID does not exist" containerID="a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018" Jul 22 11:57:21.121166 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.121082 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018"} err="failed to get container status \"a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018\": rpc error: code = NotFound desc = could not find container \"a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018\": container with ID starting with a79be9d9ed7beb59745ff3ab1789466ed13331ad5f8fff872943929b79f5e018 not found: ID does not exist" Jul 22 11:57:21.121166 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.121107 2579 scope.go:117] "RemoveContainer" containerID="c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b" Jul 22 11:57:21.121389 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:57:21.121373 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b\": container with ID starting with c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b not found: ID does not exist" containerID="c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b" Jul 22 11:57:21.121451 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:21.121396 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b"} err="failed to get container status \"c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b\": rpc error: code = NotFound desc = could not find container \"c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b\": container with ID starting with c443d7713f78d84adcecc9860abd35c509e0fe11b8d2c28bc02fb87913398d1b not found: ID does not exist" Jul 22 11:57:22.205647 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:22.205615 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163ee316-d300-4a21-964b-7ef70815f18e" path="/var/lib/kubelet/pods/163ee316-d300-4a21-964b-7ef70815f18e/volumes" Jul 22 11:57:22.206124 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:22.206110 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" path="/var/lib/kubelet/pods/7b2f4790-8e41-43e8-acb0-06b9b53dab65/volumes" Jul 22 11:57:27.151679 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.151641 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt"] Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.151952 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36e0c665-84f6-45b7-b2d5-ed603e2732ab" containerName="redis" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.151963 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="36e0c665-84f6-45b7-b2d5-ed603e2732ab" containerName="redis" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.151972 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="feast-init" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.151979 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="feast-init" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.151986 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="feast-apply" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.151992 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="feast-apply" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152005 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="registry" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152012 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="registry" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152017 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="registry" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152022 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="registry" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152033 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="feast-apply" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152038 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="feast-apply" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152046 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="feast-init" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152051 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="feast-init" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152057 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc4b953-2a31-4f79-8157-ba40eb91ba5f" containerName="manager" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152062 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc4b953-2a31-4f79-8157-ba40eb91ba5f" containerName="manager" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152067 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="online" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152072 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="online" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152081 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cfa467-b4c8-4a43-a6d7-4ddb53840612" containerName="postgres" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152085 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cfa467-b4c8-4a43-a6d7-4ddb53840612" containerName="postgres" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152090 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="online" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152096 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="online" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152139 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="registry" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152146 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cfa467-b4c8-4a43-a6d7-4ddb53840612" containerName="postgres" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152153 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="36e0c665-84f6-45b7-b2d5-ed603e2732ab" containerName="redis" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152160 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="registry" Jul 22 11:57:27.152151 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152166 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc4b953-2a31-4f79-8157-ba40eb91ba5f" containerName="manager" Jul 22 11:57:27.153042 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152172 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b2f4790-8e41-43e8-acb0-06b9b53dab65" containerName="online" Jul 22 11:57:27.153042 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.152177 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="163ee316-d300-4a21-964b-7ef70815f18e" containerName="online" Jul 22 11:57:27.157455 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.157431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" Jul 22 11:57:27.159942 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.159918 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 22 11:57:27.160094 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.159920 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 22 11:57:27.160094 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.159920 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-th7ss\"" Jul 22 11:57:27.162959 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.162934 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt"] Jul 22 11:57:27.223285 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.223246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rql\" (UniqueName: \"kubernetes.io/projected/d597aa88-f19e-4465-b5f7-b164335a8901-kube-api-access-m9rql\") pod \"feast-operator-controller-manager-7495bcd-xwbmt\" (UID: \"d597aa88-f19e-4465-b5f7-b164335a8901\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" Jul 22 11:57:27.324496 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.324452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rql\" (UniqueName: \"kubernetes.io/projected/d597aa88-f19e-4465-b5f7-b164335a8901-kube-api-access-m9rql\") pod \"feast-operator-controller-manager-7495bcd-xwbmt\" (UID: \"d597aa88-f19e-4465-b5f7-b164335a8901\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" Jul 22 11:57:27.333400 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.333364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rql\" (UniqueName: \"kubernetes.io/projected/d597aa88-f19e-4465-b5f7-b164335a8901-kube-api-access-m9rql\") pod \"feast-operator-controller-manager-7495bcd-xwbmt\" (UID: \"d597aa88-f19e-4465-b5f7-b164335a8901\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" Jul 22 11:57:27.467835 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.467726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" Jul 22 11:57:27.608544 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:27.608512 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt"] Jul 22 11:57:27.611055 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:57:27.611026 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd597aa88_f19e_4465_b5f7_b164335a8901.slice/crio-6151f009c4bbc7d809472a45a21cd89c4a905ad085203875eb13eb182dc50688 WatchSource:0}: Error finding container 6151f009c4bbc7d809472a45a21cd89c4a905ad085203875eb13eb182dc50688: Status 404 returned error can't find the container with id 6151f009c4bbc7d809472a45a21cd89c4a905ad085203875eb13eb182dc50688 Jul 22 11:57:28.085792 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:28.085751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" event={"ID":"d597aa88-f19e-4465-b5f7-b164335a8901","Type":"ContainerStarted","Data":"6151f009c4bbc7d809472a45a21cd89c4a905ad085203875eb13eb182dc50688"} Jul 22 11:57:31.100776 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:31.100730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" event={"ID":"d597aa88-f19e-4465-b5f7-b164335a8901","Type":"ContainerStarted","Data":"9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed"} Jul 22 11:57:31.101223 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:31.100836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" Jul 22 11:57:31.119669 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:31.119615 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" podStartSLOduration=1.6324067549999999 podStartE2EDuration="4.119601094s" totalImagesPullingTime="2.487194339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:57:31.11812592 +0000 UTC m=+1287.417100878" watchObservedRunningTime="2026-07-22 11:57:31.119601094 +0000 UTC m=+1287.418576052" Jul 22 11:57:42.107923 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:42.107889 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" Jul 22 11:57:54.509986 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.509939 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj"] Jul 22 11:57:54.513955 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.513935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.516593 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.516575 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 22 11:57:54.517623 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.517598 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 22 11:57:54.517623 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.517614 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 22 11:57:54.517808 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.517633 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 22 11:57:54.517808 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.517696 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-2l5p7\"" Jul 22 11:57:54.517808 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.517704 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 22 11:57:54.517808 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.517636 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 22 11:57:54.531480 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.531453 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj"] Jul 22 11:57:54.659149 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.659116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.659149 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.659162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2079e85d-909b-422c-9b87-a7919c6237e9-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.659418 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.659184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4vn\" (UniqueName: \"kubernetes.io/projected/2079e85d-909b-422c-9b87-a7919c6237e9-kube-api-access-vw4vn\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.659418 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.659296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.659418 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.659314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.659418 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.659353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.760071 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.759966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.760071 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.760020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.760071 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.760069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.760412 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.760122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.760412 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.760165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2079e85d-909b-422c-9b87-a7919c6237e9-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.760412 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.760254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4vn\" (UniqueName: \"kubernetes.io/projected/2079e85d-909b-422c-9b87-a7919c6237e9-kube-api-access-vw4vn\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.760566 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.760535 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2079e85d-909b-422c-9b87-a7919c6237e9-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.762674 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.762643 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.762800 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.762724 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.762903 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.762883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.762950 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.762883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.769346 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.769327 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4vn\" (UniqueName: \"kubernetes.io/projected/2079e85d-909b-422c-9b87-a7919c6237e9-kube-api-access-vw4vn\") pod \"feast-simple-feast-setup-777d6cc998-r9nkj\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.837311 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.837268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:57:54.995156 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:54.995129 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj"] Jul 22 11:57:54.997152 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:57:54.997126 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2079e85d_909b_422c_9b87_a7919c6237e9.slice/crio-1b29228a495275eff337d9dace2ffe6b87965178e212696ff82162f2e971f1e8 WatchSource:0}: Error finding container 1b29228a495275eff337d9dace2ffe6b87965178e212696ff82162f2e971f1e8: Status 404 returned error can't find the container with id 1b29228a495275eff337d9dace2ffe6b87965178e212696ff82162f2e971f1e8 Jul 22 11:57:55.201686 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:57:55.201649 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerStarted","Data":"1b29228a495275eff337d9dace2ffe6b87965178e212696ff82162f2e971f1e8"} Jul 22 11:58:09.290016 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:09.289978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerStarted","Data":"cb593c20c06e035b0e59afe564dbe8a59b182c845b58bf176b6a0ce7b449aa6c"} Jul 22 11:58:13.311610 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:13.311575 2579 generic.go:356] "Generic (PLEG): container finished" podID="2079e85d-909b-422c-9b87-a7919c6237e9" containerID="cb593c20c06e035b0e59afe564dbe8a59b182c845b58bf176b6a0ce7b449aa6c" exitCode=0 Jul 22 11:58:13.311987 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:13.311647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerDied","Data":"cb593c20c06e035b0e59afe564dbe8a59b182c845b58bf176b6a0ce7b449aa6c"} Jul 22 11:58:14.318614 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:14.318577 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerStarted","Data":"c8a9ff7ca15efc645151c7d69db1d737bb9b48201f08e179ca482a60c7b49f9f"} Jul 22 11:58:19.347348 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:19.347248 2579 generic.go:356] "Generic (PLEG): container finished" podID="2079e85d-909b-422c-9b87-a7919c6237e9" containerID="c8a9ff7ca15efc645151c7d69db1d737bb9b48201f08e179ca482a60c7b49f9f" exitCode=0 Jul 22 11:58:19.347348 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:19.347307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerDied","Data":"c8a9ff7ca15efc645151c7d69db1d737bb9b48201f08e179ca482a60c7b49f9f"} Jul 22 11:58:20.359407 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:20.359362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerStarted","Data":"ff85bfa4e042e07a0e38509418f2c3e6470232ee82723c9fcdf87b64fbc42678"} Jul 22 11:58:20.359871 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:20.359425 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerStarted","Data":"ed20ddc17a7cc642e86d2792bc6c0bc8f8a10cdb8e166c09679d9c3a2a8fc52c"} Jul 22 11:58:20.359871 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:20.359444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerStarted","Data":"b10e28bcd2d26758a5a5625c8ce7e605e22efd9e686e61c852d7cfc55f231fd3"} Jul 22 11:58:20.359871 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:20.359461 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerStarted","Data":"b2880a9e2995f45cf917cc066173a6205dcf47f7fef94be99238e081074f5af2"} Jul 22 11:58:20.394862 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:20.394786 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podStartSLOduration=12.953481092 podStartE2EDuration="26.394763064s" totalImagesPullingTime="13.441281972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:58:20.391230928 +0000 UTC m=+1336.690205891" watchObservedRunningTime="2026-07-22 11:58:20.394763064 +0000 UTC m=+1336.693738023" Jul 22 11:58:21.838706 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:21.838593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:21.838706 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:21.838654 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:21.838706 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:21.838675 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:21.838706 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:21.838693 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:21.840551 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:21.840511 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.39:8816: connect: connection refused" Jul 22 11:58:21.840691 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:21.840539 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Jul 22 11:58:21.840691 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:21.840551 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Jul 22 11:58:21.840691 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:21.840621 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.39:6571: connect: connection refused" Jul 22 11:58:24.837390 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:24.837345 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:24.838018 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:24.837987 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:24.838018 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:24.837631 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.39:6571: connect: connection refused" Jul 22 11:58:24.838262 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:24.837652 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Jul 22 11:58:24.838262 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:24.837684 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.39:8816: connect: connection refused" Jul 22 11:58:24.838262 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:24.838025 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:24.838262 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:24.837693 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Jul 22 11:58:24.838262 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:24.838126 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:27.838532 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:27.838443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:27.839004 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:27.838871 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:27.839004 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:27.838998 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:27.839726 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:27.839699 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:27.839868 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:27.839846 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:27.841321 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:27.841304 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:27.844047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:27.844025 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:58:28.408153 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:58:28.408122 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:59:31.210044 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.210010 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj"] Jul 22 11:59:31.210640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.210269 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="registry" containerID="cri-o://b2880a9e2995f45cf917cc066173a6205dcf47f7fef94be99238e081074f5af2" gracePeriod=30 Jul 22 11:59:31.210640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.210317 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="offline" containerID="cri-o://ed20ddc17a7cc642e86d2792bc6c0bc8f8a10cdb8e166c09679d9c3a2a8fc52c" gracePeriod=30 Jul 22 11:59:31.210640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.210335 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="ui" containerID="cri-o://ff85bfa4e042e07a0e38509418f2c3e6470232ee82723c9fcdf87b64fbc42678" gracePeriod=30 Jul 22 11:59:31.210640 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.210334 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="online" containerID="cri-o://b10e28bcd2d26758a5a5625c8ce7e605e22efd9e686e61c852d7cfc55f231fd3" gracePeriod=30 Jul 22 11:59:31.582216 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.582103 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn"] Jul 22 11:59:31.585759 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.585739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.588702 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.588679 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-q2m4c\"" Jul 22 11:59:31.603209 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.603162 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn"] Jul 22 11:59:31.701864 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.701833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.702057 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.701872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.702057 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.701910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.702057 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.701938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk2np\" (UniqueName: \"kubernetes.io/projected/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-kube-api-access-wk2np\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.702057 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.702001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.702057 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.702045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.802635 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.802599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.802815 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.802657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.802815 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.802700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.802815 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.802728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.802815 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.802763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.802815 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.802794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk2np\" (UniqueName: \"kubernetes.io/projected/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-kube-api-access-wk2np\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.803171 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:59:31.802886 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 22 11:59:31.803171 ip-10-0-132-210 kubenswrapper[2579]: E0722 11:59:31.802959 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-offline-tls podName:438e9ab6-86d9-4d4e-9072-51d3b25e1a7c nodeName:}" failed. No retries permitted until 2026-07-22 11:59:32.302938028 +0000 UTC m=+1408.601912983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-offline-tls") pod "feast-simple-feast-setup-777d6cc998-xnrcn" (UID: "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c") : secret "feast-simple-feast-setup-offline-tls" not found Jul 22 11:59:31.803171 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.803108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.805496 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.805465 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.805693 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.805671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.805748 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.805725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:31.812858 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:31.812831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk2np\" (UniqueName: \"kubernetes.io/projected/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-kube-api-access-wk2np\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:32.307677 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:32.307637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:32.310248 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:32.310187 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-xnrcn\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:32.497059 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:32.497016 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:32.637713 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:32.637685 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn"] Jul 22 11:59:32.638730 ip-10-0-132-210 kubenswrapper[2579]: W0722 11:59:32.638699 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod438e9ab6_86d9_4d4e_9072_51d3b25e1a7c.slice/crio-91bad6eb7f82f3c92e681f99c789adfbf2235fd89f7196731c42e96f57b8c384 WatchSource:0}: Error finding container 91bad6eb7f82f3c92e681f99c789adfbf2235fd89f7196731c42e96f57b8c384: Status 404 returned error can't find the container with id 91bad6eb7f82f3c92e681f99c789adfbf2235fd89f7196731c42e96f57b8c384 Jul 22 11:59:32.695522 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:32.695495 2579 generic.go:356] "Generic (PLEG): container finished" podID="2079e85d-909b-422c-9b87-a7919c6237e9" containerID="ff85bfa4e042e07a0e38509418f2c3e6470232ee82723c9fcdf87b64fbc42678" exitCode=0 Jul 22 11:59:32.695522 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:32.695519 2579 generic.go:356] "Generic (PLEG): container finished" podID="2079e85d-909b-422c-9b87-a7919c6237e9" containerID="b10e28bcd2d26758a5a5625c8ce7e605e22efd9e686e61c852d7cfc55f231fd3" exitCode=0 Jul 22 11:59:32.695709 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:32.695565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerDied","Data":"ff85bfa4e042e07a0e38509418f2c3e6470232ee82723c9fcdf87b64fbc42678"} Jul 22 11:59:32.695709 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:32.695606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerDied","Data":"b10e28bcd2d26758a5a5625c8ce7e605e22efd9e686e61c852d7cfc55f231fd3"} Jul 22 11:59:32.697117 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:32.697088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerStarted","Data":"91bad6eb7f82f3c92e681f99c789adfbf2235fd89f7196731c42e96f57b8c384"} Jul 22 11:59:33.703774 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:33.703738 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerStarted","Data":"e9716ebd5153de4ae8bb8224a092f9c1282263f54afaaaff0e8c873a3675cc86"} Jul 22 11:59:37.722076 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:37.722040 2579 generic.go:356] "Generic (PLEG): container finished" podID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerID="e9716ebd5153de4ae8bb8224a092f9c1282263f54afaaaff0e8c873a3675cc86" exitCode=0 Jul 22 11:59:37.722474 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:37.722112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerDied","Data":"e9716ebd5153de4ae8bb8224a092f9c1282263f54afaaaff0e8c873a3675cc86"} Jul 22 11:59:37.839820 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:37.839779 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Jul 22 11:59:38.405530 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:38.405485 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Jul 22 11:59:38.728311 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:38.728224 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerStarted","Data":"d1155b80ea326c5595551dcd1638395a52d29a033fd881a985ca7c031035a511"} Jul 22 11:59:43.750679 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:43.750643 2579 generic.go:356] "Generic (PLEG): container finished" podID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerID="d1155b80ea326c5595551dcd1638395a52d29a033fd881a985ca7c031035a511" exitCode=0 Jul 22 11:59:43.751141 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:43.750721 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerDied","Data":"d1155b80ea326c5595551dcd1638395a52d29a033fd881a985ca7c031035a511"} Jul 22 11:59:44.762586 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:44.762543 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerStarted","Data":"3bb20ee1be21dd91cc37fc0f41e416e0bde1f1d16d9efa1ba14e71f29dd37ac7"} Jul 22 11:59:44.763292 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:44.762608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerStarted","Data":"4a8701ac59d6d326a621bd0f147d4cc4c0e2922bd846a308fb3d5b685a0b49e4"} Jul 22 11:59:44.763292 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:44.762630 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerStarted","Data":"8d431763c4aee6df850b9e48862ef79d967bb239e1ede2396679d443668f80e4"} Jul 22 11:59:44.763292 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:44.762649 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerStarted","Data":"7e1fe1db0827c0d5c4c96a0086f3c0cae5f6b9c6c01c98f37da7ea6d1af386fe"} Jul 22 11:59:44.792680 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:44.792606 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podStartSLOduration=13.792582391 podStartE2EDuration="13.792582391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 11:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 11:59:44.789766883 +0000 UTC m=+1421.088741879" watchObservedRunningTime="2026-07-22 11:59:44.792582391 +0000 UTC m=+1421.091557349" Jul 22 11:59:47.497831 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:47.497788 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:47.498419 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:47.498396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:47.498575 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:47.498428 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:47.498575 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:47.498446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:47.499980 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:47.499949 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Jul 22 11:59:47.500209 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:47.499958 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Jul 22 11:59:47.500300 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:47.499994 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.40:8816: connect: connection refused" Jul 22 11:59:47.500300 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:47.500037 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.40:6571: connect: connection refused" Jul 22 11:59:47.839860 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:47.839763 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Jul 22 11:59:48.405305 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:48.405251 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Jul 22 11:59:50.498324 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.498295 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.498791 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.498464 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.498791 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.498711 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.499047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.498995 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.499047 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.499043 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.499270 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.499062 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.499357 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.499326 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.499444 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.499404 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.499517 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.499500 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.503268 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.503253 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.801065 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.800986 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:50.804311 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:50.804283 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 11:59:57.840170 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:57.840004 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" Jul 22 11:59:57.840170 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:57.840164 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 11:59:58.405558 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:58.405510 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" Jul 22 11:59:58.405728 ip-10-0-132-210 kubenswrapper[2579]: I0722 11:59:58.405640 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 12:00:01.853382 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.853354 2579 generic.go:356] "Generic (PLEG): container finished" podID="2079e85d-909b-422c-9b87-a7919c6237e9" containerID="ed20ddc17a7cc642e86d2792bc6c0bc8f8a10cdb8e166c09679d9c3a2a8fc52c" exitCode=137 Jul 22 12:00:01.853382 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.853378 2579 generic.go:356] "Generic (PLEG): container finished" podID="2079e85d-909b-422c-9b87-a7919c6237e9" containerID="b2880a9e2995f45cf917cc066173a6205dcf47f7fef94be99238e081074f5af2" exitCode=137 Jul 22 12:00:01.853755 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.853406 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerDied","Data":"ed20ddc17a7cc642e86d2792bc6c0bc8f8a10cdb8e166c09679d9c3a2a8fc52c"} Jul 22 12:00:01.853755 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.853435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerDied","Data":"b2880a9e2995f45cf917cc066173a6205dcf47f7fef94be99238e081074f5af2"} Jul 22 12:00:01.870990 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.870964 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 12:00:01.993219 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.993101 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-ui-tls\") pod \"2079e85d-909b-422c-9b87-a7919c6237e9\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " Jul 22 12:00:01.993219 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.993173 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2079e85d-909b-422c-9b87-a7919c6237e9-feast-data\") pod \"2079e85d-909b-422c-9b87-a7919c6237e9\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " Jul 22 12:00:01.993431 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.993227 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-offline-tls\") pod \"2079e85d-909b-422c-9b87-a7919c6237e9\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " Jul 22 12:00:01.993431 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.993251 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-online-tls\") pod \"2079e85d-909b-422c-9b87-a7919c6237e9\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " Jul 22 12:00:01.993431 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.993292 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw4vn\" (UniqueName: \"kubernetes.io/projected/2079e85d-909b-422c-9b87-a7919c6237e9-kube-api-access-vw4vn\") pod \"2079e85d-909b-422c-9b87-a7919c6237e9\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " Jul 22 12:00:01.993431 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.993319 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-registry-tls\") pod \"2079e85d-909b-422c-9b87-a7919c6237e9\" (UID: \"2079e85d-909b-422c-9b87-a7919c6237e9\") " Jul 22 12:00:01.993797 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.993768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2079e85d-909b-422c-9b87-a7919c6237e9-feast-data" (OuterVolumeSpecName: "feast-data") pod "2079e85d-909b-422c-9b87-a7919c6237e9" (UID: "2079e85d-909b-422c-9b87-a7919c6237e9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:00:01.995571 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.995542 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-online-tls" (OuterVolumeSpecName: "online-tls") pod "2079e85d-909b-422c-9b87-a7919c6237e9" (UID: "2079e85d-909b-422c-9b87-a7919c6237e9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:00:01.995668 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.995578 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2079e85d-909b-422c-9b87-a7919c6237e9" (UID: "2079e85d-909b-422c-9b87-a7919c6237e9"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:00:01.995668 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.995604 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2079e85d-909b-422c-9b87-a7919c6237e9" (UID: "2079e85d-909b-422c-9b87-a7919c6237e9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:00:01.995790 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.995709 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2079e85d-909b-422c-9b87-a7919c6237e9" (UID: "2079e85d-909b-422c-9b87-a7919c6237e9"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:00:01.995845 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:01.995795 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2079e85d-909b-422c-9b87-a7919c6237e9-kube-api-access-vw4vn" (OuterVolumeSpecName: "kube-api-access-vw4vn") pod "2079e85d-909b-422c-9b87-a7919c6237e9" (UID: "2079e85d-909b-422c-9b87-a7919c6237e9"). InnerVolumeSpecName "kube-api-access-vw4vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:00:02.094019 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.093988 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-offline-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:00:02.094301 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.094033 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:00:02.094301 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.094043 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw4vn\" (UniqueName: \"kubernetes.io/projected/2079e85d-909b-422c-9b87-a7919c6237e9-kube-api-access-vw4vn\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:00:02.094301 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.094053 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:00:02.094301 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.094061 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2079e85d-909b-422c-9b87-a7919c6237e9-ui-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:00:02.094301 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.094069 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2079e85d-909b-422c-9b87-a7919c6237e9-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:00:02.860857 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.860821 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" Jul 22 12:00:02.860857 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.860830 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj" event={"ID":"2079e85d-909b-422c-9b87-a7919c6237e9","Type":"ContainerDied","Data":"1b29228a495275eff337d9dace2ffe6b87965178e212696ff82162f2e971f1e8"} Jul 22 12:00:02.861402 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.860893 2579 scope.go:117] "RemoveContainer" containerID="ff85bfa4e042e07a0e38509418f2c3e6470232ee82723c9fcdf87b64fbc42678" Jul 22 12:00:02.869538 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.869509 2579 scope.go:117] "RemoveContainer" containerID="ed20ddc17a7cc642e86d2792bc6c0bc8f8a10cdb8e166c09679d9c3a2a8fc52c" Jul 22 12:00:02.877077 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.877059 2579 scope.go:117] "RemoveContainer" containerID="b10e28bcd2d26758a5a5625c8ce7e605e22efd9e686e61c852d7cfc55f231fd3" Jul 22 12:00:02.879946 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.879926 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj"] Jul 22 12:00:02.885257 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.885235 2579 scope.go:117] "RemoveContainer" containerID="b2880a9e2995f45cf917cc066173a6205dcf47f7fef94be99238e081074f5af2" Jul 22 12:00:02.885997 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.885977 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-r9nkj"] Jul 22 12:00:02.892729 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.892712 2579 scope.go:117] "RemoveContainer" containerID="c8a9ff7ca15efc645151c7d69db1d737bb9b48201f08e179ca482a60c7b49f9f" Jul 22 12:00:02.899833 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:02.899819 2579 scope.go:117] "RemoveContainer" containerID="cb593c20c06e035b0e59afe564dbe8a59b182c845b58bf176b6a0ce7b449aa6c" Jul 22 12:00:04.212607 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:04.212562 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" path="/var/lib/kubelet/pods/2079e85d-909b-422c-9b87-a7919c6237e9/volumes" Jul 22 12:00:53.422702 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.422663 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95"] Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423010 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="feast-apply" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423022 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="feast-apply" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423039 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="online" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423047 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="online" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423056 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="ui" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423061 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="ui" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423074 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="offline" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423079 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="offline" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423087 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="registry" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423092 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="registry" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423102 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="feast-init" Jul 22 12:00:53.423125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423108 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="feast-init" Jul 22 12:00:53.423536 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423170 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="registry" Jul 22 12:00:53.423536 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423179 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="ui" Jul 22 12:00:53.423536 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423186 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="offline" Jul 22 12:00:53.423536 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.423210 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2079e85d-909b-422c-9b87-a7919c6237e9" containerName="online" Jul 22 12:00:53.426626 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.426608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.431150 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.431123 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 22 12:00:53.431310 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.431165 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 22 12:00:53.431360 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.431311 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 22 12:00:53.431695 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.431677 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 22 12:00:53.431755 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.431703 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 22 12:00:53.431755 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.431722 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-6l7kn\"" Jul 22 12:00:53.431845 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.431757 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 22 12:00:53.443710 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.443692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95"] Jul 22 12:00:53.500640 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.500610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x6db\" (UniqueName: \"kubernetes.io/projected/e36bfe86-2f7e-4842-83e1-62f545e4c966-kube-api-access-6x6db\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.500805 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.500650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e36bfe86-2f7e-4842-83e1-62f545e4c966-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.500805 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.500717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.500805 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.500732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e36bfe86-2f7e-4842-83e1-62f545e4c966-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.500805 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.500754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.500964 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.500815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.602114 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.602083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x6db\" (UniqueName: \"kubernetes.io/projected/e36bfe86-2f7e-4842-83e1-62f545e4c966-kube-api-access-6x6db\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.602337 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.602124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e36bfe86-2f7e-4842-83e1-62f545e4c966-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.602337 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.602304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.602487 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.602339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e36bfe86-2f7e-4842-83e1-62f545e4c966-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.602487 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.602380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.602487 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.602409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.602642 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:00:53.602491 2579 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 22 12:00:53.602642 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:00:53.602512 2579 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 22 12:00:53.602642 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:00:53.602563 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-offline-tls podName:e36bfe86-2f7e-4842-83e1-62f545e4c966 nodeName:}" failed. No retries permitted until 2026-07-22 12:00:54.102545175 +0000 UTC m=+1490.401520111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-offline-tls") pod "feast-simple-feast-remote-setup-db6955c6c-5jz95" (UID: "e36bfe86-2f7e-4842-83e1-62f545e4c966") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 22 12:00:53.602642 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:00:53.602577 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-ui-tls podName:e36bfe86-2f7e-4842-83e1-62f545e4c966 nodeName:}" failed. No retries permitted until 2026-07-22 12:00:54.102571184 +0000 UTC m=+1490.401546119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-ui-tls") pod "feast-simple-feast-remote-setup-db6955c6c-5jz95" (UID: "e36bfe86-2f7e-4842-83e1-62f545e4c966") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 22 12:00:53.602844 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.602765 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e36bfe86-2f7e-4842-83e1-62f545e4c966-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.602844 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.602828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e36bfe86-2f7e-4842-83e1-62f545e4c966-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.604717 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.604699 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:53.612513 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:53.612479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x6db\" (UniqueName: \"kubernetes.io/projected/e36bfe86-2f7e-4842-83e1-62f545e4c966-kube-api-access-6x6db\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:54.106529 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:54.106493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:54.106529 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:54.106533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:54.108995 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:54.108970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:54.109090 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:54.109057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-5jz95\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:54.336393 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:54.336349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:00:54.475297 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:54.475273 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95"] Jul 22 12:00:54.477130 ip-10-0-132-210 kubenswrapper[2579]: W0722 12:00:54.477103 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode36bfe86_2f7e_4842_83e1_62f545e4c966.slice/crio-72c5b3cce349dc74ab9a9145ef7147fbe1e93e97b5edde9befb05ef2dae52e0f WatchSource:0}: Error finding container 72c5b3cce349dc74ab9a9145ef7147fbe1e93e97b5edde9befb05ef2dae52e0f: Status 404 returned error can't find the container with id 72c5b3cce349dc74ab9a9145ef7147fbe1e93e97b5edde9befb05ef2dae52e0f Jul 22 12:00:55.106938 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:55.106898 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerStarted","Data":"97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71"} Jul 22 12:00:55.107133 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:00:55.106948 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerStarted","Data":"72c5b3cce349dc74ab9a9145ef7147fbe1e93e97b5edde9befb05ef2dae52e0f"} Jul 22 12:01:00.132245 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:00.132183 2579 generic.go:356] "Generic (PLEG): container finished" podID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerID="97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71" exitCode=0 Jul 22 12:01:00.132655 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:00.132253 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerDied","Data":"97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71"} Jul 22 12:01:01.139466 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:01.139421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerStarted","Data":"552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9"} Jul 22 12:01:04.232261 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:04.232230 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 12:01:04.234125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:04.234104 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 12:01:07.169974 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:07.169940 2579 generic.go:356] "Generic (PLEG): container finished" podID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerID="552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9" exitCode=0 Jul 22 12:01:07.170420 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:07.170020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerDied","Data":"552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9"} Jul 22 12:01:08.181779 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:08.181736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerStarted","Data":"59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8"} Jul 22 12:01:08.182346 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:08.181804 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerStarted","Data":"4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3"} Jul 22 12:01:08.182346 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:08.181828 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerStarted","Data":"8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03"} Jul 22 12:01:08.216935 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:08.216855 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podStartSLOduration=15.216831901 podStartE2EDuration="15.216831901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:01:08.214292447 +0000 UTC m=+1504.513267406" watchObservedRunningTime="2026-07-22 12:01:08.216831901 +0000 UTC m=+1504.515806860" Jul 22 12:01:09.336853 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:09.336815 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:09.337392 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:09.336971 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:09.337392 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:09.336995 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:09.338905 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:09.338870 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.41:8816: connect: connection refused" Jul 22 12:01:09.338905 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:09.338892 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Jul 22 12:01:09.339061 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:09.338893 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Jul 22 12:01:12.337494 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:12.337443 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.41:8816: connect: connection refused" Jul 22 12:01:12.338504 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:12.338480 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Jul 22 12:01:12.338694 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:12.338442 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Jul 22 12:01:14.336478 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:14.336435 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:14.336898 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:14.336493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:14.336898 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:14.336505 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:15.337557 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:15.337521 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:15.338055 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:15.337622 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:15.338269 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:15.338247 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:15.338343 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:15.338302 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:15.344219 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:15.344170 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:01:16.231252 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:01:16.231187 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:02:18.917396 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:18.917356 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95"] Jul 22 12:02:18.918005 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:18.917593 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="online" containerID="cri-o://8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03" gracePeriod=30 Jul 22 12:02:18.918005 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:18.917638 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="offline" containerID="cri-o://4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3" gracePeriod=30 Jul 22 12:02:18.918005 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:18.917651 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="ui" containerID="cri-o://59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8" gracePeriod=30 Jul 22 12:02:19.143540 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:19.143493 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn"] Jul 22 12:02:19.143871 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:19.143813 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="registry" containerID="cri-o://7e1fe1db0827c0d5c4c96a0086f3c0cae5f6b9c6c01c98f37da7ea6d1af386fe" gracePeriod=30 Jul 22 12:02:19.144125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:19.143865 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="offline" containerID="cri-o://4a8701ac59d6d326a621bd0f147d4cc4c0e2922bd846a308fb3d5b685a0b49e4" gracePeriod=30 Jul 22 12:02:19.144277 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:19.143872 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="online" containerID="cri-o://8d431763c4aee6df850b9e48862ef79d967bb239e1ede2396679d443668f80e4" gracePeriod=30 Jul 22 12:02:19.144334 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:19.143885 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="ui" containerID="cri-o://3bb20ee1be21dd91cc37fc0f41e416e0bde1f1d16d9efa1ba14e71f29dd37ac7" gracePeriod=30 Jul 22 12:02:20.499342 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:20.499291 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Jul 22 12:02:20.523171 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:20.523137 2579 generic.go:356] "Generic (PLEG): container finished" podID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerID="3bb20ee1be21dd91cc37fc0f41e416e0bde1f1d16d9efa1ba14e71f29dd37ac7" exitCode=0 Jul 22 12:02:20.523171 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:20.523166 2579 generic.go:356] "Generic (PLEG): container finished" podID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerID="8d431763c4aee6df850b9e48862ef79d967bb239e1ede2396679d443668f80e4" exitCode=0 Jul 22 12:02:20.523400 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:20.523220 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerDied","Data":"3bb20ee1be21dd91cc37fc0f41e416e0bde1f1d16d9efa1ba14e71f29dd37ac7"} Jul 22 12:02:20.523400 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:20.523277 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerDied","Data":"8d431763c4aee6df850b9e48862ef79d967bb239e1ede2396679d443668f80e4"} Jul 22 12:02:20.525610 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:20.525588 2579 generic.go:356] "Generic (PLEG): container finished" podID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerID="59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8" exitCode=0 Jul 22 12:02:20.525610 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:20.525604 2579 generic.go:356] "Generic (PLEG): container finished" podID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerID="8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03" exitCode=0 Jul 22 12:02:20.525770 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:20.525642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerDied","Data":"59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8"} Jul 22 12:02:20.525770 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:20.525668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerDied","Data":"8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03"} Jul 22 12:02:20.801854 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:20.801757 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Jul 22 12:02:25.338550 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:25.338504 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Jul 22 12:02:26.227576 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:26.227528 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Jul 22 12:02:30.499331 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:30.499249 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Jul 22 12:02:30.801969 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:30.801875 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Jul 22 12:02:35.338385 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:35.338334 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Jul 22 12:02:36.227218 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:36.227157 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Jul 22 12:02:40.499814 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:40.499760 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" Jul 22 12:02:40.500331 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:40.499917 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 12:02:40.801991 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:40.801886 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" Jul 22 12:02:40.802163 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:40.802025 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 12:02:45.338865 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:45.338814 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.41:8443: connect: connection refused" Jul 22 12:02:45.339296 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:45.338970 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:02:46.226795 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:46.226757 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="online" probeResult="failure" output="Get \"https://10.132.0.41:6567/health\": dial tcp 10.132.0.41:6567: connect: connection refused" Jul 22 12:02:46.226973 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:46.226857 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:02:49.100240 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.100216 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:02:49.212332 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.212308 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x6db\" (UniqueName: \"kubernetes.io/projected/e36bfe86-2f7e-4842-83e1-62f545e4c966-kube-api-access-6x6db\") pod \"e36bfe86-2f7e-4842-83e1-62f545e4c966\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " Jul 22 12:02:49.212475 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.212373 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-ui-tls\") pod \"e36bfe86-2f7e-4842-83e1-62f545e4c966\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " Jul 22 12:02:49.212475 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.212412 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e36bfe86-2f7e-4842-83e1-62f545e4c966-feast-data\") pod \"e36bfe86-2f7e-4842-83e1-62f545e4c966\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " Jul 22 12:02:49.212475 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.212458 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e36bfe86-2f7e-4842-83e1-62f545e4c966-registry-tls\") pod \"e36bfe86-2f7e-4842-83e1-62f545e4c966\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " Jul 22 12:02:49.212635 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.212480 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-offline-tls\") pod \"e36bfe86-2f7e-4842-83e1-62f545e4c966\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " Jul 22 12:02:49.212635 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.212530 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-online-tls\") pod \"e36bfe86-2f7e-4842-83e1-62f545e4c966\" (UID: \"e36bfe86-2f7e-4842-83e1-62f545e4c966\") " Jul 22 12:02:49.212923 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.212892 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36bfe86-2f7e-4842-83e1-62f545e4c966-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e36bfe86-2f7e-4842-83e1-62f545e4c966" (UID: "e36bfe86-2f7e-4842-83e1-62f545e4c966"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:02:49.213074 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.213032 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36bfe86-2f7e-4842-83e1-62f545e4c966-feast-data" (OuterVolumeSpecName: "feast-data") pod "e36bfe86-2f7e-4842-83e1-62f545e4c966" (UID: "e36bfe86-2f7e-4842-83e1-62f545e4c966"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:02:49.214469 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.214443 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36bfe86-2f7e-4842-83e1-62f545e4c966-kube-api-access-6x6db" (OuterVolumeSpecName: "kube-api-access-6x6db") pod "e36bfe86-2f7e-4842-83e1-62f545e4c966" (UID: "e36bfe86-2f7e-4842-83e1-62f545e4c966"). InnerVolumeSpecName "kube-api-access-6x6db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:02:49.214562 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.214539 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e36bfe86-2f7e-4842-83e1-62f545e4c966" (UID: "e36bfe86-2f7e-4842-83e1-62f545e4c966"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:02:49.214749 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.214731 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e36bfe86-2f7e-4842-83e1-62f545e4c966" (UID: "e36bfe86-2f7e-4842-83e1-62f545e4c966"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:02:49.214804 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.214745 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-online-tls" (OuterVolumeSpecName: "online-tls") pod "e36bfe86-2f7e-4842-83e1-62f545e4c966" (UID: "e36bfe86-2f7e-4842-83e1-62f545e4c966"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:02:49.313966 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.313931 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:49.313966 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.313959 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x6db\" (UniqueName: \"kubernetes.io/projected/e36bfe86-2f7e-4842-83e1-62f545e4c966-kube-api-access-6x6db\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:49.313966 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.313971 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-ui-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:49.314284 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.313984 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e36bfe86-2f7e-4842-83e1-62f545e4c966-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:49.314284 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.313996 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e36bfe86-2f7e-4842-83e1-62f545e4c966-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:49.314284 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.314006 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e36bfe86-2f7e-4842-83e1-62f545e4c966-offline-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:49.659583 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.659550 2579 generic.go:356] "Generic (PLEG): container finished" podID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerID="4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3" exitCode=137 Jul 22 12:02:49.659762 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.659630 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" Jul 22 12:02:49.659762 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.659629 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerDied","Data":"4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3"} Jul 22 12:02:49.659870 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.659768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95" event={"ID":"e36bfe86-2f7e-4842-83e1-62f545e4c966","Type":"ContainerDied","Data":"72c5b3cce349dc74ab9a9145ef7147fbe1e93e97b5edde9befb05ef2dae52e0f"} Jul 22 12:02:49.659870 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.659800 2579 scope.go:117] "RemoveContainer" containerID="59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8" Jul 22 12:02:49.663568 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.663543 2579 generic.go:356] "Generic (PLEG): container finished" podID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerID="4a8701ac59d6d326a621bd0f147d4cc4c0e2922bd846a308fb3d5b685a0b49e4" exitCode=137 Jul 22 12:02:49.663568 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.663566 2579 generic.go:356] "Generic (PLEG): container finished" podID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerID="7e1fe1db0827c0d5c4c96a0086f3c0cae5f6b9c6c01c98f37da7ea6d1af386fe" exitCode=137 Jul 22 12:02:49.663705 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.663590 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerDied","Data":"4a8701ac59d6d326a621bd0f147d4cc4c0e2922bd846a308fb3d5b685a0b49e4"} Jul 22 12:02:49.663705 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.663646 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerDied","Data":"7e1fe1db0827c0d5c4c96a0086f3c0cae5f6b9c6c01c98f37da7ea6d1af386fe"} Jul 22 12:02:49.671470 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.671425 2579 scope.go:117] "RemoveContainer" containerID="4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3" Jul 22 12:02:49.681122 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.681098 2579 scope.go:117] "RemoveContainer" containerID="8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03" Jul 22 12:02:49.688526 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.688504 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95"] Jul 22 12:02:49.689004 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.688984 2579 scope.go:117] "RemoveContainer" containerID="552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9" Jul 22 12:02:49.692055 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.692033 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-5jz95"] Jul 22 12:02:49.698415 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.698391 2579 scope.go:117] "RemoveContainer" containerID="97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71" Jul 22 12:02:49.705541 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.705523 2579 scope.go:117] "RemoveContainer" containerID="59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8" Jul 22 12:02:49.705767 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:02:49.705750 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8\": container with ID starting with 59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8 not found: ID does not exist" containerID="59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8" Jul 22 12:02:49.705851 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.705775 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8"} err="failed to get container status \"59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8\": rpc error: code = NotFound desc = could not find container \"59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8\": container with ID starting with 59638e64b27ddd46c0ff04ef011a2bdd5b7699666bdc6fb2a0f413085da9cea8 not found: ID does not exist" Jul 22 12:02:49.705851 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.705797 2579 scope.go:117] "RemoveContainer" containerID="4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3" Jul 22 12:02:49.706027 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:02:49.706006 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3\": container with ID starting with 4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3 not found: ID does not exist" containerID="4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3" Jul 22 12:02:49.706080 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.706032 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3"} err="failed to get container status \"4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3\": rpc error: code = NotFound desc = could not find container \"4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3\": container with ID starting with 4f8a4b7e3f330a0c17faea202940c9043d764d08c5b659ef864c7978412df1d3 not found: ID does not exist" Jul 22 12:02:49.706080 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.706048 2579 scope.go:117] "RemoveContainer" containerID="8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03" Jul 22 12:02:49.706348 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:02:49.706329 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03\": container with ID starting with 8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03 not found: ID does not exist" containerID="8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03" Jul 22 12:02:49.706402 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.706353 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03"} err="failed to get container status \"8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03\": rpc error: code = NotFound desc = could not find container \"8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03\": container with ID starting with 8721fd7a67d8c998194ddaa3019818a6b516293dfa0a776eb4bf0b5dab287e03 not found: ID does not exist" Jul 22 12:02:49.706402 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.706369 2579 scope.go:117] "RemoveContainer" containerID="552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9" Jul 22 12:02:49.706605 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:02:49.706587 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9\": container with ID starting with 552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9 not found: ID does not exist" containerID="552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9" Jul 22 12:02:49.706663 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.706608 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9"} err="failed to get container status \"552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9\": rpc error: code = NotFound desc = could not find container \"552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9\": container with ID starting with 552a8cf2ca431db59557a75ef020413e437f12119f032aa48bf2db0db0b6eee9 not found: ID does not exist" Jul 22 12:02:49.706663 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.706622 2579 scope.go:117] "RemoveContainer" containerID="97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71" Jul 22 12:02:49.706822 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:02:49.706806 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71\": container with ID starting with 97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71 not found: ID does not exist" containerID="97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71" Jul 22 12:02:49.706881 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.706823 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71"} err="failed to get container status \"97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71\": rpc error: code = NotFound desc = could not find container \"97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71\": container with ID starting with 97bf98eed723948fc8d8db8c04c3db1809b9febee981d818243c36c35cab8e71 not found: ID does not exist" Jul 22 12:02:49.784913 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.784888 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 12:02:49.921068 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.920973 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk2np\" (UniqueName: \"kubernetes.io/projected/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-kube-api-access-wk2np\") pod \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " Jul 22 12:02:49.921068 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.921057 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-ui-tls\") pod \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " Jul 22 12:02:49.921325 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.921099 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-online-tls\") pod \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " Jul 22 12:02:49.921325 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.921128 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-feast-data\") pod \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " Jul 22 12:02:49.921325 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.921158 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-offline-tls\") pod \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " Jul 22 12:02:49.921325 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.921181 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-registry-tls\") pod \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\" (UID: \"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c\") " Jul 22 12:02:49.921775 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.921743 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-feast-data" (OuterVolumeSpecName: "feast-data") pod "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" (UID: "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:02:49.923236 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.923209 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" (UID: "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:02:49.923236 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.923226 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" (UID: "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:02:49.923367 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.923243 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-kube-api-access-wk2np" (OuterVolumeSpecName: "kube-api-access-wk2np") pod "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" (UID: "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c"). InnerVolumeSpecName "kube-api-access-wk2np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:02:49.923562 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.923539 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-online-tls" (OuterVolumeSpecName: "online-tls") pod "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" (UID: "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:02:49.923562 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:49.923549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" (UID: "438e9ab6-86d9-4d4e-9072-51d3b25e1a7c"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:02:50.022465 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.022436 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:50.022465 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.022461 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-offline-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:50.022465 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.022470 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:50.022724 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.022479 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk2np\" (UniqueName: \"kubernetes.io/projected/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-kube-api-access-wk2np\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:50.022724 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.022490 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-ui-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:50.022724 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.022498 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:02:50.205875 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.205796 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" path="/var/lib/kubelet/pods/e36bfe86-2f7e-4842-83e1-62f545e4c966/volumes" Jul 22 12:02:50.671168 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.671139 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" Jul 22 12:02:50.671168 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.671149 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn" event={"ID":"438e9ab6-86d9-4d4e-9072-51d3b25e1a7c","Type":"ContainerDied","Data":"91bad6eb7f82f3c92e681f99c789adfbf2235fd89f7196731c42e96f57b8c384"} Jul 22 12:02:50.671443 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.671221 2579 scope.go:117] "RemoveContainer" containerID="3bb20ee1be21dd91cc37fc0f41e416e0bde1f1d16d9efa1ba14e71f29dd37ac7" Jul 22 12:02:50.679493 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.679476 2579 scope.go:117] "RemoveContainer" containerID="4a8701ac59d6d326a621bd0f147d4cc4c0e2922bd846a308fb3d5b685a0b49e4" Jul 22 12:02:50.686779 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.686759 2579 scope.go:117] "RemoveContainer" containerID="8d431763c4aee6df850b9e48862ef79d967bb239e1ede2396679d443668f80e4" Jul 22 12:02:50.693771 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.693751 2579 scope.go:117] "RemoveContainer" containerID="7e1fe1db0827c0d5c4c96a0086f3c0cae5f6b9c6c01c98f37da7ea6d1af386fe" Jul 22 12:02:50.696833 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.696807 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn"] Jul 22 12:02:50.698863 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.698842 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-xnrcn"] Jul 22 12:02:50.701990 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.701975 2579 scope.go:117] "RemoveContainer" containerID="d1155b80ea326c5595551dcd1638395a52d29a033fd881a985ca7c031035a511" Jul 22 12:02:50.708564 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:50.708550 2579 scope.go:117] "RemoveContainer" containerID="e9716ebd5153de4ae8bb8224a092f9c1282263f54afaaaff0e8c873a3675cc86" Jul 22 12:02:52.205156 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:02:52.205122 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" path="/var/lib/kubelet/pods/438e9ab6-86d9-4d4e-9072-51d3b25e1a7c/volumes" Jul 22 12:03:08.309472 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.309437 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt"] Jul 22 12:03:08.311887 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.309648 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" podUID="d597aa88-f19e-4465-b5f7-b164335a8901" containerName="manager" containerID="cri-o://9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed" gracePeriod=10 Jul 22 12:03:08.554455 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.554426 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" Jul 22 12:03:08.679821 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.679784 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9rql\" (UniqueName: \"kubernetes.io/projected/d597aa88-f19e-4465-b5f7-b164335a8901-kube-api-access-m9rql\") pod \"d597aa88-f19e-4465-b5f7-b164335a8901\" (UID: \"d597aa88-f19e-4465-b5f7-b164335a8901\") " Jul 22 12:03:08.681993 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.681954 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d597aa88-f19e-4465-b5f7-b164335a8901-kube-api-access-m9rql" (OuterVolumeSpecName: "kube-api-access-m9rql") pod "d597aa88-f19e-4465-b5f7-b164335a8901" (UID: "d597aa88-f19e-4465-b5f7-b164335a8901"). InnerVolumeSpecName "kube-api-access-m9rql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:03:08.751637 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.751603 2579 generic.go:356] "Generic (PLEG): container finished" podID="d597aa88-f19e-4465-b5f7-b164335a8901" containerID="9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed" exitCode=0 Jul 22 12:03:08.751817 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.751647 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" Jul 22 12:03:08.751817 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.751688 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" event={"ID":"d597aa88-f19e-4465-b5f7-b164335a8901","Type":"ContainerDied","Data":"9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed"} Jul 22 12:03:08.751817 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.751741 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt" event={"ID":"d597aa88-f19e-4465-b5f7-b164335a8901","Type":"ContainerDied","Data":"6151f009c4bbc7d809472a45a21cd89c4a905ad085203875eb13eb182dc50688"} Jul 22 12:03:08.751817 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.751766 2579 scope.go:117] "RemoveContainer" containerID="9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed" Jul 22 12:03:08.760160 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.760145 2579 scope.go:117] "RemoveContainer" containerID="9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed" Jul 22 12:03:08.760424 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:03:08.760407 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed\": container with ID starting with 9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed not found: ID does not exist" containerID="9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed" Jul 22 12:03:08.760474 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.760432 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed"} err="failed to get container status \"9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed\": rpc error: code = NotFound desc = could not find container \"9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed\": container with ID starting with 9b2752a31e5677b7311d4278757818b29a42f65fd9b1c839959cd48f6422d5ed not found: ID does not exist" Jul 22 12:03:08.773939 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.773918 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt"] Jul 22 12:03:08.778647 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.778628 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-xwbmt"] Jul 22 12:03:08.780459 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:08.780429 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9rql\" (UniqueName: \"kubernetes.io/projected/d597aa88-f19e-4465-b5f7-b164335a8901-kube-api-access-m9rql\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:03:10.206074 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:10.206039 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d597aa88-f19e-4465-b5f7-b164335a8901" path="/var/lib/kubelet/pods/d597aa88-f19e-4465-b5f7-b164335a8901/volumes" Jul 22 12:03:13.156422 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156380 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l"] Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156704 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d597aa88-f19e-4465-b5f7-b164335a8901" containerName="manager" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156715 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d597aa88-f19e-4465-b5f7-b164335a8901" containerName="manager" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156723 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="ui" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156729 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="ui" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156736 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="ui" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156743 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="ui" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156751 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="online" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156757 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="online" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156765 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="offline" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156770 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="offline" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156778 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="registry" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156783 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="registry" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156792 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="feast-init" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156797 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="feast-init" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156804 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="offline" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156809 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="offline" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156813 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="online" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156818 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="online" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156826 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="feast-apply" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156831 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="feast-apply" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156840 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="feast-init" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156845 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="feast-init" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156851 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="feast-apply" Jul 22 12:03:13.156846 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156856 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="feast-apply" Jul 22 12:03:13.157585 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156902 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="registry" Jul 22 12:03:13.157585 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156909 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="online" Jul 22 12:03:13.157585 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156915 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="online" Jul 22 12:03:13.157585 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156919 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="offline" Jul 22 12:03:13.157585 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156926 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="ui" Jul 22 12:03:13.157585 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156932 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36bfe86-2f7e-4842-83e1-62f545e4c966" containerName="ui" Jul 22 12:03:13.157585 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156937 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="438e9ab6-86d9-4d4e-9072-51d3b25e1a7c" containerName="offline" Jul 22 12:03:13.157585 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.156941 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d597aa88-f19e-4465-b5f7-b164335a8901" containerName="manager" Jul 22 12:03:13.161205 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.161178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l" Jul 22 12:03:13.163771 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.163744 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 22 12:03:13.164695 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.164677 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-th7ss\"" Jul 22 12:03:13.164839 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.164677 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 22 12:03:13.168817 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.168782 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l"] Jul 22 12:03:13.322864 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.322819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kftsn\" (UniqueName: \"kubernetes.io/projected/e78e2cfe-c8cc-4afd-90cf-cbc0f816ead7-kube-api-access-kftsn\") pod \"feast-operator-controller-manager-7495bcd-dfr9l\" (UID: \"e78e2cfe-c8cc-4afd-90cf-cbc0f816ead7\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l" Jul 22 12:03:13.424225 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.424113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftsn\" (UniqueName: \"kubernetes.io/projected/e78e2cfe-c8cc-4afd-90cf-cbc0f816ead7-kube-api-access-kftsn\") pod \"feast-operator-controller-manager-7495bcd-dfr9l\" (UID: \"e78e2cfe-c8cc-4afd-90cf-cbc0f816ead7\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l" Jul 22 12:03:13.433095 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.433064 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kftsn\" (UniqueName: \"kubernetes.io/projected/e78e2cfe-c8cc-4afd-90cf-cbc0f816ead7-kube-api-access-kftsn\") pod \"feast-operator-controller-manager-7495bcd-dfr9l\" (UID: \"e78e2cfe-c8cc-4afd-90cf-cbc0f816ead7\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l" Jul 22 12:03:13.472917 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.472875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l" Jul 22 12:03:13.612566 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.612541 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l"] Jul 22 12:03:13.614881 ip-10-0-132-210 kubenswrapper[2579]: W0722 12:03:13.614851 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78e2cfe_c8cc_4afd_90cf_cbc0f816ead7.slice/crio-1486cb23f86baaeee94e6c4e40fd580d7738e3fb2a6f6bcb034c7b5b91674d41 WatchSource:0}: Error finding container 1486cb23f86baaeee94e6c4e40fd580d7738e3fb2a6f6bcb034c7b5b91674d41: Status 404 returned error can't find the container with id 1486cb23f86baaeee94e6c4e40fd580d7738e3fb2a6f6bcb034c7b5b91674d41 Jul 22 12:03:13.780040 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.780001 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l" event={"ID":"e78e2cfe-c8cc-4afd-90cf-cbc0f816ead7","Type":"ContainerStarted","Data":"a57e8e07687fdddf4bb5eb311287dd5b7416f92401a68bd826ff16b458778d1b"} Jul 22 12:03:13.780245 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.780056 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l" event={"ID":"e78e2cfe-c8cc-4afd-90cf-cbc0f816ead7","Type":"ContainerStarted","Data":"1486cb23f86baaeee94e6c4e40fd580d7738e3fb2a6f6bcb034c7b5b91674d41"} Jul 22 12:03:13.780245 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.780138 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l" Jul 22 12:03:13.799321 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:13.799242 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l" podStartSLOduration=0.799221927 podStartE2EDuration="799.221927ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:03:13.797799564 +0000 UTC m=+1630.096774569" watchObservedRunningTime="2026-07-22 12:03:13.799221927 +0000 UTC m=+1630.098196898" Jul 22 12:03:24.786078 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:24.786049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-dfr9l" Jul 22 12:03:29.921679 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.921639 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m"] Jul 22 12:03:29.925591 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.925573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:29.928329 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.928305 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 22 12:03:29.928329 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.928322 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 22 12:03:29.928598 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.928579 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-bt6rx\"" Jul 22 12:03:29.928733 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.928717 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 22 12:03:29.929349 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.929334 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 22 12:03:29.929434 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.929356 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 22 12:03:29.929434 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.929356 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 22 12:03:29.946998 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.946974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:29.947121 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.947002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:29.947121 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.947031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:29.947121 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.947048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:29.947300 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.947134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc6cf3f-e112-4296-a68d-7aa836823838-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:29.947300 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.947178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnq2\" (UniqueName: \"kubernetes.io/projected/bbc6cf3f-e112-4296-a68d-7aa836823838-kube-api-access-hvnq2\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:29.951979 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:29.951945 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m"] Jul 22 12:03:30.047678 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.047637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.047678 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.047676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.047934 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.047717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.047934 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.047733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.047934 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.047757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc6cf3f-e112-4296-a68d-7aa836823838-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.047934 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.047779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnq2\" (UniqueName: \"kubernetes.io/projected/bbc6cf3f-e112-4296-a68d-7aa836823838-kube-api-access-hvnq2\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.047934 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:03:30.047859 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 22 12:03:30.047934 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:03:30.047937 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-ui-tls podName:bbc6cf3f-e112-4296-a68d-7aa836823838 nodeName:}" failed. No retries permitted until 2026-07-22 12:03:30.547920377 +0000 UTC m=+1646.846895313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-ui-tls") pod "feast-simple-feast-setup-777d6cc998-8959m" (UID: "bbc6cf3f-e112-4296-a68d-7aa836823838") : secret "feast-simple-feast-setup-ui-tls" not found Jul 22 12:03:30.048296 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:03:30.047864 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 22 12:03:30.048296 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:03:30.048047 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-registry-tls podName:bbc6cf3f-e112-4296-a68d-7aa836823838 nodeName:}" failed. No retries permitted until 2026-07-22 12:03:30.548023165 +0000 UTC m=+1646.846998106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-registry-tls") pod "feast-simple-feast-setup-777d6cc998-8959m" (UID: "bbc6cf3f-e112-4296-a68d-7aa836823838") : secret "feast-simple-feast-setup-registry-tls" not found Jul 22 12:03:30.048296 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.048268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc6cf3f-e112-4296-a68d-7aa836823838-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.050281 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.050253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.050395 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.050326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.056258 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.056237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnq2\" (UniqueName: \"kubernetes.io/projected/bbc6cf3f-e112-4296-a68d-7aa836823838-kube-api-access-hvnq2\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.552511 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.552474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.552511 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.552513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.554912 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.554877 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.554912 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.554902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-8959m\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.846264 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.846143 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:30.979222 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:30.979183 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m"] Jul 22 12:03:30.981585 ip-10-0-132-210 kubenswrapper[2579]: W0722 12:03:30.981557 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbc6cf3f_e112_4296_a68d_7aa836823838.slice/crio-8c2e48eb35fa954fccf8826a3b2df0f524e731b75094625dbbfd687c411b695f WatchSource:0}: Error finding container 8c2e48eb35fa954fccf8826a3b2df0f524e731b75094625dbbfd687c411b695f: Status 404 returned error can't find the container with id 8c2e48eb35fa954fccf8826a3b2df0f524e731b75094625dbbfd687c411b695f Jul 22 12:03:31.855022 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:31.854982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerStarted","Data":"53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d"} Jul 22 12:03:31.855022 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:31.855025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerStarted","Data":"8c2e48eb35fa954fccf8826a3b2df0f524e731b75094625dbbfd687c411b695f"} Jul 22 12:03:35.875159 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:35.875122 2579 generic.go:356] "Generic (PLEG): container finished" podID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerID="53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d" exitCode=0 Jul 22 12:03:35.875571 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:35.875206 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerDied","Data":"53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d"} Jul 22 12:03:36.881266 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:36.881227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerStarted","Data":"c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3"} Jul 22 12:03:41.903815 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:41.903778 2579 generic.go:356] "Generic (PLEG): container finished" podID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerID="c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3" exitCode=0 Jul 22 12:03:41.904313 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:41.903827 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerDied","Data":"c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3"} Jul 22 12:03:42.915288 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:42.915235 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerStarted","Data":"00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2"} Jul 22 12:03:42.915802 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:42.915309 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerStarted","Data":"636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc"} Jul 22 12:03:42.915802 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:42.915329 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerStarted","Data":"e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884"} Jul 22 12:03:42.915802 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:42.915351 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerStarted","Data":"3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc"} Jul 22 12:03:42.945039 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:42.944976 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podStartSLOduration=13.944950732 podStartE2EDuration="13.944950732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:03:42.942818544 +0000 UTC m=+1659.241793538" watchObservedRunningTime="2026-07-22 12:03:42.944950732 +0000 UTC m=+1659.243925690" Jul 22 12:03:45.847438 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:45.847395 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:45.847912 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:45.847465 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:45.847912 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:45.847487 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:45.847912 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:45.847504 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:45.849436 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:45.849395 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.43:8443: connect: connection refused" Jul 22 12:03:45.849436 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:45.849413 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.43:8816: connect: connection refused" Jul 22 12:03:45.849645 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:45.849420 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.43:6571: connect: connection refused" Jul 22 12:03:45.849645 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:45.849477 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="online" probeResult="failure" output="Get \"https://10.132.0.43:6567/health\": dial tcp 10.132.0.43:6567: connect: connection refused" Jul 22 12:03:48.848540 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.848502 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.849033 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.848656 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.849033 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.848863 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.849246 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.849220 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.849424 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.849262 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.849424 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.849283 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.849617 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.849601 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.849689 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.849676 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.851320 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.851305 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.853470 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.853452 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.951615 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.951588 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:03:48.954893 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:03:48.954873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:04:51.457244 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.457181 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m"] Jul 22 12:04:51.460755 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.460692 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="registry" containerID="cri-o://3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc" gracePeriod=30 Jul 22 12:04:51.461352 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.461292 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="ui" containerID="cri-o://00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2" gracePeriod=30 Jul 22 12:04:51.461885 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.461533 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="online" containerID="cri-o://e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884" gracePeriod=30 Jul 22 12:04:51.461995 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.461970 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="offline" containerID="cri-o://636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc" gracePeriod=30 Jul 22 12:04:51.777356 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.777259 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q"] Jul 22 12:04:51.781782 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.781754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.784386 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.784350 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-d2m4n\"" Jul 22 12:04:51.793712 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.793680 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q"] Jul 22 12:04:51.862381 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.862345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.862381 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.862381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.862676 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.862469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.862676 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.862526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7cg\" (UniqueName: \"kubernetes.io/projected/d640a04d-7b6d-4031-867b-73bb074c7caa-kube-api-access-vd7cg\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.862676 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.862577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.862676 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.862619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d640a04d-7b6d-4031-867b-73bb074c7caa-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.963028 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.962990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.963156 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.963041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.963156 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.963106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.963156 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.963136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7cg\" (UniqueName: \"kubernetes.io/projected/d640a04d-7b6d-4031-867b-73bb074c7caa-kube-api-access-vd7cg\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.963358 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.963182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.963358 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.963275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d640a04d-7b6d-4031-867b-73bb074c7caa-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.963358 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:04:51.963312 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 22 12:04:51.963509 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:04:51.963390 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-online-tls podName:d640a04d-7b6d-4031-867b-73bb074c7caa nodeName:}" failed. No retries permitted until 2026-07-22 12:04:52.463368957 +0000 UTC m=+1728.762343930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-online-tls") pod "feast-simple-feast-setup-777d6cc998-c8w6q" (UID: "d640a04d-7b6d-4031-867b-73bb074c7caa") : secret "feast-simple-feast-setup-online-tls" not found Jul 22 12:04:51.963693 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.963651 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d640a04d-7b6d-4031-867b-73bb074c7caa-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.963801 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:04:51.963707 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 22 12:04:51.963859 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:04:51.963812 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-ui-tls podName:d640a04d-7b6d-4031-867b-73bb074c7caa nodeName:}" failed. No retries permitted until 2026-07-22 12:04:52.463780125 +0000 UTC m=+1728.762755082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-ui-tls") pod "feast-simple-feast-setup-777d6cc998-c8w6q" (UID: "d640a04d-7b6d-4031-867b-73bb074c7caa") : secret "feast-simple-feast-setup-ui-tls" not found Jul 22 12:04:51.965849 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.965822 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.965976 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.965918 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:51.972640 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:51.972613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7cg\" (UniqueName: \"kubernetes.io/projected/d640a04d-7b6d-4031-867b-73bb074c7caa-kube-api-access-vd7cg\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:52.235122 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:52.235090 2579 generic.go:356] "Generic (PLEG): container finished" podID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerID="00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2" exitCode=0 Jul 22 12:04:52.235122 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:52.235115 2579 generic.go:356] "Generic (PLEG): container finished" podID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerID="e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884" exitCode=0 Jul 22 12:04:52.235354 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:52.235164 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerDied","Data":"00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2"} Jul 22 12:04:52.235354 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:52.235230 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerDied","Data":"e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884"} Jul 22 12:04:52.467496 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:52.467442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:52.467888 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:52.467547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:52.469941 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:52.469908 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:52.470075 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:52.470048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-c8w6q\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:52.692262 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:52.692219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:04:52.833662 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:52.833629 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q"] Jul 22 12:04:52.834853 ip-10-0-132-210 kubenswrapper[2579]: W0722 12:04:52.834828 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd640a04d_7b6d_4031_867b_73bb074c7caa.slice/crio-cfb389ea66728686fd7077ad5b278d1885ff36c29f2b8372849291a5941b1436 WatchSource:0}: Error finding container cfb389ea66728686fd7077ad5b278d1885ff36c29f2b8372849291a5941b1436: Status 404 returned error can't find the container with id cfb389ea66728686fd7077ad5b278d1885ff36c29f2b8372849291a5941b1436 Jul 22 12:04:53.241708 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:53.241665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerStarted","Data":"2a7256eb995c260df4de7cf2619d3c3279c9d1c25fda91a1ab9f8499ee44de3f"} Jul 22 12:04:53.241708 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:53.241715 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerStarted","Data":"cfb389ea66728686fd7077ad5b278d1885ff36c29f2b8372849291a5941b1436"} Jul 22 12:04:58.265311 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:58.265278 2579 generic.go:356] "Generic (PLEG): container finished" podID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerID="2a7256eb995c260df4de7cf2619d3c3279c9d1c25fda91a1ab9f8499ee44de3f" exitCode=0 Jul 22 12:04:58.265831 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:58.265360 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerDied","Data":"2a7256eb995c260df4de7cf2619d3c3279c9d1c25fda91a1ab9f8499ee44de3f"} Jul 22 12:04:58.849497 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:58.849435 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.43:8443: connect: connection refused" Jul 22 12:04:58.952569 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:58.952519 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="online" probeResult="failure" output="Get \"https://10.132.0.43:6567/health\": dial tcp 10.132.0.43:6567: connect: connection refused" Jul 22 12:04:59.272653 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:04:59.272616 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerStarted","Data":"bba1b85f40f12ba7dd191aae95fe0396b00f9e028618a21d8a0aa0ceac56fd04"} Jul 22 12:05:04.298633 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:04.298598 2579 generic.go:356] "Generic (PLEG): container finished" podID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerID="bba1b85f40f12ba7dd191aae95fe0396b00f9e028618a21d8a0aa0ceac56fd04" exitCode=0 Jul 22 12:05:04.298633 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:04.298635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerDied","Data":"bba1b85f40f12ba7dd191aae95fe0396b00f9e028618a21d8a0aa0ceac56fd04"} Jul 22 12:05:05.310181 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:05.310141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerStarted","Data":"a3a21a17a85af9d33ce973a5039f27baf956a59069c734a006727ba08c927518"} Jul 22 12:05:05.310790 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:05.310764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerStarted","Data":"03db949a18b00a12d2164e341bdd36d78346745c19f6adfc8adf3983d9cf11f7"} Jul 22 12:05:05.310914 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:05.310896 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerStarted","Data":"15a93f120088e951837ecd594927fd3680f02bbeb2a6b4be04e4e8dbea6dc23d"} Jul 22 12:05:05.311016 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:05.311003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerStarted","Data":"d31069ba6dbbdab2409adce06808d773c07d3154bd921169c0402d3ca8ed9e02"} Jul 22 12:05:05.338919 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:05.338855 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podStartSLOduration=14.338832454 podStartE2EDuration="14.338832454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:05:05.33774245 +0000 UTC m=+1741.636717447" watchObservedRunningTime="2026-07-22 12:05:05.338832454 +0000 UTC m=+1741.637807415" Jul 22 12:05:07.693319 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:07.693275 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:07.693838 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:07.693345 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:07.693838 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:07.693368 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:07.693838 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:07.693386 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:07.695674 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:07.695522 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.44:8816: connect: connection refused" Jul 22 12:05:07.695674 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:07.695572 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.44:8443: connect: connection refused" Jul 22 12:05:07.695674 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:07.695532 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.44:6571: connect: connection refused" Jul 22 12:05:07.695674 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:07.695614 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": dial tcp 10.132.0.44:6567: connect: connection refused" Jul 22 12:05:08.849989 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:08.849923 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.43:8443: connect: connection refused" Jul 22 12:05:08.952789 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:08.952732 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="online" probeResult="failure" output="Get \"https://10.132.0.43:6567/health\": dial tcp 10.132.0.43:6567: connect: connection refused" Jul 22 12:05:10.693180 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:10.693111 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.44:8443: connect: connection refused" Jul 22 12:05:10.693963 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:10.693936 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:10.694246 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:10.694226 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:10.694620 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:10.694596 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:10.694746 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:10.694638 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:10.694746 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:10.694739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:10.694848 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:10.694810 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:10.699899 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:10.699874 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:11.348477 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:11.348444 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:11.352155 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:11.352128 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:12.693156 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:12.693114 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:13.693929 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:13.693898 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:14.361865 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:14.361831 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:05:18.849779 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:18.849726 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.43:8443: connect: connection refused" Jul 22 12:05:18.850324 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:18.849884 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:05:18.952480 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:18.952436 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="online" probeResult="failure" output="Get \"https://10.132.0.43:6567/health\": dial tcp 10.132.0.43:6567: connect: connection refused" Jul 22 12:05:18.952669 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:18.952575 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:05:22.116903 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.116876 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:05:22.152177 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.152140 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc6cf3f-e112-4296-a68d-7aa836823838-feast-data\") pod \"bbc6cf3f-e112-4296-a68d-7aa836823838\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " Jul 22 12:05:22.152391 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.152246 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-offline-tls\") pod \"bbc6cf3f-e112-4296-a68d-7aa836823838\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " Jul 22 12:05:22.152391 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.152264 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-online-tls\") pod \"bbc6cf3f-e112-4296-a68d-7aa836823838\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " Jul 22 12:05:22.152391 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.152318 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvnq2\" (UniqueName: \"kubernetes.io/projected/bbc6cf3f-e112-4296-a68d-7aa836823838-kube-api-access-hvnq2\") pod \"bbc6cf3f-e112-4296-a68d-7aa836823838\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " Jul 22 12:05:22.152391 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.152364 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-registry-tls\") pod \"bbc6cf3f-e112-4296-a68d-7aa836823838\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " Jul 22 12:05:22.152391 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.152390 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-ui-tls\") pod \"bbc6cf3f-e112-4296-a68d-7aa836823838\" (UID: \"bbc6cf3f-e112-4296-a68d-7aa836823838\") " Jul 22 12:05:22.152764 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.152736 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc6cf3f-e112-4296-a68d-7aa836823838-feast-data" (OuterVolumeSpecName: "feast-data") pod "bbc6cf3f-e112-4296-a68d-7aa836823838" (UID: "bbc6cf3f-e112-4296-a68d-7aa836823838"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:05:22.154612 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.154576 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bbc6cf3f-e112-4296-a68d-7aa836823838" (UID: "bbc6cf3f-e112-4296-a68d-7aa836823838"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:05:22.154612 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.154596 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-online-tls" (OuterVolumeSpecName: "online-tls") pod "bbc6cf3f-e112-4296-a68d-7aa836823838" (UID: "bbc6cf3f-e112-4296-a68d-7aa836823838"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:05:22.154787 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.154612 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bbc6cf3f-e112-4296-a68d-7aa836823838" (UID: "bbc6cf3f-e112-4296-a68d-7aa836823838"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:05:22.154858 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.154834 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "bbc6cf3f-e112-4296-a68d-7aa836823838" (UID: "bbc6cf3f-e112-4296-a68d-7aa836823838"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:05:22.155053 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.155036 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc6cf3f-e112-4296-a68d-7aa836823838-kube-api-access-hvnq2" (OuterVolumeSpecName: "kube-api-access-hvnq2") pod "bbc6cf3f-e112-4296-a68d-7aa836823838" (UID: "bbc6cf3f-e112-4296-a68d-7aa836823838"). InnerVolumeSpecName "kube-api-access-hvnq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:05:22.253889 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.253799 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-offline-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:05:22.253889 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.253851 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:05:22.253889 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.253861 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvnq2\" (UniqueName: \"kubernetes.io/projected/bbc6cf3f-e112-4296-a68d-7aa836823838-kube-api-access-hvnq2\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:05:22.253889 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.253873 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:05:22.253889 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.253882 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc6cf3f-e112-4296-a68d-7aa836823838-ui-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:05:22.253889 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.253889 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc6cf3f-e112-4296-a68d-7aa836823838-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:05:22.400129 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.400088 2579 generic.go:356] "Generic (PLEG): container finished" podID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerID="636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc" exitCode=137 Jul 22 12:05:22.400129 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.400124 2579 generic.go:356] "Generic (PLEG): container finished" podID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerID="3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc" exitCode=137 Jul 22 12:05:22.400374 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.400160 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" Jul 22 12:05:22.400374 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.400167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerDied","Data":"636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc"} Jul 22 12:05:22.400374 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.400241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerDied","Data":"3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc"} Jul 22 12:05:22.400374 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.400257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m" event={"ID":"bbc6cf3f-e112-4296-a68d-7aa836823838","Type":"ContainerDied","Data":"8c2e48eb35fa954fccf8826a3b2df0f524e731b75094625dbbfd687c411b695f"} Jul 22 12:05:22.400374 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.400277 2579 scope.go:117] "RemoveContainer" containerID="00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2" Jul 22 12:05:22.409350 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.409326 2579 scope.go:117] "RemoveContainer" containerID="636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc" Jul 22 12:05:22.417374 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.417351 2579 scope.go:117] "RemoveContainer" containerID="e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884" Jul 22 12:05:22.420311 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.420285 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m"] Jul 22 12:05:22.424464 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.424440 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-8959m"] Jul 22 12:05:22.427036 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.427019 2579 scope.go:117] "RemoveContainer" containerID="3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc" Jul 22 12:05:22.435167 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.435147 2579 scope.go:117] "RemoveContainer" containerID="c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3" Jul 22 12:05:22.443005 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.442985 2579 scope.go:117] "RemoveContainer" containerID="53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d" Jul 22 12:05:22.450786 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.450766 2579 scope.go:117] "RemoveContainer" containerID="00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2" Jul 22 12:05:22.451068 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:05:22.451044 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2\": container with ID starting with 00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2 not found: ID does not exist" containerID="00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2" Jul 22 12:05:22.451157 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.451074 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2"} err="failed to get container status \"00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2\": rpc error: code = NotFound desc = could not find container \"00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2\": container with ID starting with 00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2 not found: ID does not exist" Jul 22 12:05:22.451157 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.451094 2579 scope.go:117] "RemoveContainer" containerID="636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc" Jul 22 12:05:22.451365 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:05:22.451348 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc\": container with ID starting with 636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc not found: ID does not exist" containerID="636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc" Jul 22 12:05:22.451422 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.451370 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc"} err="failed to get container status \"636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc\": rpc error: code = NotFound desc = could not find container \"636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc\": container with ID starting with 636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc not found: ID does not exist" Jul 22 12:05:22.451422 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.451384 2579 scope.go:117] "RemoveContainer" containerID="e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884" Jul 22 12:05:22.451629 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:05:22.451612 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884\": container with ID starting with e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884 not found: ID does not exist" containerID="e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884" Jul 22 12:05:22.451664 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.451636 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884"} err="failed to get container status \"e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884\": rpc error: code = NotFound desc = could not find container \"e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884\": container with ID starting with e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884 not found: ID does not exist" Jul 22 12:05:22.451664 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.451654 2579 scope.go:117] "RemoveContainer" containerID="3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc" Jul 22 12:05:22.451865 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:05:22.451845 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc\": container with ID starting with 3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc not found: ID does not exist" containerID="3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc" Jul 22 12:05:22.451900 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.451872 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc"} err="failed to get container status \"3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc\": rpc error: code = NotFound desc = could not find container \"3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc\": container with ID starting with 3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc not found: ID does not exist" Jul 22 12:05:22.451900 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.451887 2579 scope.go:117] "RemoveContainer" containerID="c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3" Jul 22 12:05:22.452120 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:05:22.452103 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3\": container with ID starting with c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3 not found: ID does not exist" containerID="c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3" Jul 22 12:05:22.452156 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.452125 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3"} err="failed to get container status \"c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3\": rpc error: code = NotFound desc = could not find container \"c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3\": container with ID starting with c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3 not found: ID does not exist" Jul 22 12:05:22.452156 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.452139 2579 scope.go:117] "RemoveContainer" containerID="53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d" Jul 22 12:05:22.452491 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:05:22.452467 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d\": container with ID starting with 53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d not found: ID does not exist" containerID="53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d" Jul 22 12:05:22.452558 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.452494 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d"} err="failed to get container status \"53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d\": rpc error: code = NotFound desc = could not find container \"53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d\": container with ID starting with 53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d not found: ID does not exist" Jul 22 12:05:22.452558 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.452512 2579 scope.go:117] "RemoveContainer" containerID="00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2" Jul 22 12:05:22.452720 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.452704 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2"} err="failed to get container status \"00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2\": rpc error: code = NotFound desc = could not find container \"00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2\": container with ID starting with 00df132a8952da497a7d7bc7e02ca31728acc84aed7503491ae2b86697bd69f2 not found: ID does not exist" Jul 22 12:05:22.452761 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.452720 2579 scope.go:117] "RemoveContainer" containerID="636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc" Jul 22 12:05:22.452901 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.452883 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc"} err="failed to get container status \"636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc\": rpc error: code = NotFound desc = could not find container \"636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc\": container with ID starting with 636a4311a1c431ad0e47ec189540cd52d317920849f8581e6a9cd894b07dddfc not found: ID does not exist" Jul 22 12:05:22.452939 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.452901 2579 scope.go:117] "RemoveContainer" containerID="e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884" Jul 22 12:05:22.453102 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.453086 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884"} err="failed to get container status \"e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884\": rpc error: code = NotFound desc = could not find container \"e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884\": container with ID starting with e544cb9c3e1f27b57d8540aa4274151a9bf4d315352d7ba8079ae8bd9cb11884 not found: ID does not exist" Jul 22 12:05:22.453140 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.453104 2579 scope.go:117] "RemoveContainer" containerID="3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc" Jul 22 12:05:22.453349 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.453332 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc"} err="failed to get container status \"3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc\": rpc error: code = NotFound desc = could not find container \"3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc\": container with ID starting with 3851aa4aaacfe6dc303a0fa028df573fae826e24eb27c40ac82843c1428307bc not found: ID does not exist" Jul 22 12:05:22.453398 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.453350 2579 scope.go:117] "RemoveContainer" containerID="c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3" Jul 22 12:05:22.453600 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.453577 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3"} err="failed to get container status \"c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3\": rpc error: code = NotFound desc = could not find container \"c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3\": container with ID starting with c256d6e9f72c34fb7fad3dda7cf7bce3542338782a9f1cf687e22ca75b81aae3 not found: ID does not exist" Jul 22 12:05:22.453643 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.453602 2579 scope.go:117] "RemoveContainer" containerID="53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d" Jul 22 12:05:22.453809 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:22.453788 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d"} err="failed to get container status \"53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d\": rpc error: code = NotFound desc = could not find container \"53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d\": container with ID starting with 53fde9b69e72b16aae2952e0a9e6221bcfe7c8b168e89dcda88408899dde133d not found: ID does not exist" Jul 22 12:05:24.207256 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:05:24.207222 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" path="/var/lib/kubelet/pods/bbc6cf3f-e112-4296-a68d-7aa836823838/volumes" Jul 22 12:06:04.260351 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:04.260321 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 12:06:04.262821 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:04.262798 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 12:06:17.351820 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.351779 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q"] Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352105 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="offline" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352116 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="offline" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352124 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="ui" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352129 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="ui" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352137 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="online" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352142 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="online" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352153 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="feast-init" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352158 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="feast-init" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352164 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="feast-apply" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352169 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="feast-apply" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352186 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="registry" Jul 22 12:06:17.352229 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352207 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="registry" Jul 22 12:06:17.352613 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352263 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="registry" Jul 22 12:06:17.352613 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352272 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="ui" Jul 22 12:06:17.352613 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352280 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="online" Jul 22 12:06:17.352613 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.352285 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc6cf3f-e112-4296-a68d-7aa836823838" containerName="offline" Jul 22 12:06:17.356683 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.356666 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.359263 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.359236 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 22 12:06:17.359429 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.359277 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 22 12:06:17.359429 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.359312 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 22 12:06:17.359429 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.359428 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 22 12:06:17.359633 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.359432 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-h6wk9\"" Jul 22 12:06:17.359633 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.359430 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 22 12:06:17.359753 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.359719 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 22 12:06:17.371154 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.371131 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q"] Jul 22 12:06:17.427688 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.427652 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.427861 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.427708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.427861 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.427749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.427861 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.427767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.427861 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.427786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.428020 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.427869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47pbz\" (UniqueName: \"kubernetes.io/projected/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-kube-api-access-47pbz\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.528819 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.528783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.529015 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.528850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.529015 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.528871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.529015 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.528888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.529015 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.528907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.529015 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.528942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47pbz\" (UniqueName: \"kubernetes.io/projected/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-kube-api-access-47pbz\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.529438 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.529410 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.529781 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.529751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.531474 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.531452 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.531564 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.531492 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.531624 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.531605 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.537392 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.537367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47pbz\" (UniqueName: \"kubernetes.io/projected/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-kube-api-access-47pbz\") pod \"feast-simple-feast-remote-setup-db6955c6c-rnz2q\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.666070 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.666041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:17.803405 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:17.803372 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q"] Jul 22 12:06:17.806478 ip-10-0-132-210 kubenswrapper[2579]: W0722 12:06:17.806445 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a4a9224_4ca9_49ba_8b92_b7d2382edebd.slice/crio-737aa2be188ae4ce6b8c7ac9bcb7535d4b1f7bfb6901ec825ac01f7d831dda83 WatchSource:0}: Error finding container 737aa2be188ae4ce6b8c7ac9bcb7535d4b1f7bfb6901ec825ac01f7d831dda83: Status 404 returned error can't find the container with id 737aa2be188ae4ce6b8c7ac9bcb7535d4b1f7bfb6901ec825ac01f7d831dda83 Jul 22 12:06:18.645162 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:18.645127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerStarted","Data":"fb2700b499d09ce1c309a10408b63b3c531d7319377af707f1016003f3554978"} Jul 22 12:06:18.645597 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:18.645172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerStarted","Data":"737aa2be188ae4ce6b8c7ac9bcb7535d4b1f7bfb6901ec825ac01f7d831dda83"} Jul 22 12:06:22.666443 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:22.666411 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerID="fb2700b499d09ce1c309a10408b63b3c531d7319377af707f1016003f3554978" exitCode=0 Jul 22 12:06:22.666804 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:22.666490 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerDied","Data":"fb2700b499d09ce1c309a10408b63b3c531d7319377af707f1016003f3554978"} Jul 22 12:06:23.673442 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:23.673394 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerStarted","Data":"3eeeb7f209d6fdccb45bd768c0d1d79c3ae0d3ef34266c5fac3a862d20ed274e"} Jul 22 12:06:28.697291 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:28.697254 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerID="3eeeb7f209d6fdccb45bd768c0d1d79c3ae0d3ef34266c5fac3a862d20ed274e" exitCode=0 Jul 22 12:06:28.697690 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:28.697324 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerDied","Data":"3eeeb7f209d6fdccb45bd768c0d1d79c3ae0d3ef34266c5fac3a862d20ed274e"} Jul 22 12:06:29.706478 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:29.706435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerStarted","Data":"2c609ef74ed4b047cd677ee62d0dc69daee6c6fedef2b5dc94c43fc457f26ae1"} Jul 22 12:06:29.706867 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:29.706503 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerStarted","Data":"051df837cab399b722f3faa797933b2bcb40a44d0f077ff79800b907cba77390"} Jul 22 12:06:29.706867 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:29.706528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerStarted","Data":"9926a9891625ebf199c89aa77846c003670907640cfc87f5971c8c4dbf88dba3"} Jul 22 12:06:29.731285 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:29.731230 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podStartSLOduration=12.731185239 podStartE2EDuration="12.731185239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:06:29.729955906 +0000 UTC m=+1826.028930865" watchObservedRunningTime="2026-07-22 12:06:29.731185239 +0000 UTC m=+1826.030160197" Jul 22 12:06:32.666981 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:32.666935 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:32.667494 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:32.667015 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:32.667494 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:32.667032 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:32.669023 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:32.668985 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.45:8443: connect: connection refused" Jul 22 12:06:32.669147 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:32.669014 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.45:8816: connect: connection refused" Jul 22 12:06:32.669147 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:32.669031 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="online" probeResult="failure" output="Get \"https://10.132.0.45:6567/health\": dial tcp 10.132.0.45:6567: connect: connection refused" Jul 22 12:06:35.667064 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:35.667019 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:35.667674 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:35.667510 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:35.667674 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:35.667551 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:35.667835 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:35.667817 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:35.673557 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:35.673536 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:35.741783 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:35.741755 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:35.741986 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:35.741800 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:35.742336 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:35.742312 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:06:35.746418 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:06:35.746395 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:07:36.524781 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:36.524693 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q"] Jul 22 12:07:36.525372 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:36.524973 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="online" containerID="cri-o://9926a9891625ebf199c89aa77846c003670907640cfc87f5971c8c4dbf88dba3" gracePeriod=30 Jul 22 12:07:36.525372 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:36.525053 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="ui" containerID="cri-o://2c609ef74ed4b047cd677ee62d0dc69daee6c6fedef2b5dc94c43fc457f26ae1" gracePeriod=30 Jul 22 12:07:36.525372 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:36.525058 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="offline" containerID="cri-o://051df837cab399b722f3faa797933b2bcb40a44d0f077ff79800b907cba77390" gracePeriod=30 Jul 22 12:07:36.672294 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:36.672251 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q"] Jul 22 12:07:36.672651 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:36.672559 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="registry" containerID="cri-o://d31069ba6dbbdab2409adce06808d773c07d3154bd921169c0402d3ca8ed9e02" gracePeriod=30 Jul 22 12:07:36.672801 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:36.672645 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="offline" containerID="cri-o://03db949a18b00a12d2164e341bdd36d78346745c19f6adfc8adf3983d9cf11f7" gracePeriod=30 Jul 22 12:07:36.672801 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:36.672739 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="online" containerID="cri-o://15a93f120088e951837ecd594927fd3680f02bbeb2a6b4be04e4e8dbea6dc23d" gracePeriod=30 Jul 22 12:07:36.672801 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:36.672789 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="ui" containerID="cri-o://a3a21a17a85af9d33ce973a5039f27baf956a59069c734a006727ba08c927518" gracePeriod=30 Jul 22 12:07:38.032165 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:38.032132 2579 generic.go:356] "Generic (PLEG): container finished" podID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerID="a3a21a17a85af9d33ce973a5039f27baf956a59069c734a006727ba08c927518" exitCode=0 Jul 22 12:07:38.032165 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:38.032160 2579 generic.go:356] "Generic (PLEG): container finished" podID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerID="15a93f120088e951837ecd594927fd3680f02bbeb2a6b4be04e4e8dbea6dc23d" exitCode=0 Jul 22 12:07:38.032688 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:38.032219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerDied","Data":"a3a21a17a85af9d33ce973a5039f27baf956a59069c734a006727ba08c927518"} Jul 22 12:07:38.032688 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:38.032263 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerDied","Data":"15a93f120088e951837ecd594927fd3680f02bbeb2a6b4be04e4e8dbea6dc23d"} Jul 22 12:07:38.034823 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:38.034801 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerID="2c609ef74ed4b047cd677ee62d0dc69daee6c6fedef2b5dc94c43fc457f26ae1" exitCode=0 Jul 22 12:07:38.034823 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:38.034818 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerID="9926a9891625ebf199c89aa77846c003670907640cfc87f5971c8c4dbf88dba3" exitCode=0 Jul 22 12:07:38.034973 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:38.034872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerDied","Data":"2c609ef74ed4b047cd677ee62d0dc69daee6c6fedef2b5dc94c43fc457f26ae1"} Jul 22 12:07:38.034973 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:38.034906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerDied","Data":"9926a9891625ebf199c89aa77846c003670907640cfc87f5971c8c4dbf88dba3"} Jul 22 12:07:41.349034 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:41.348993 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": dial tcp 10.132.0.44:6567: connect: connection refused" Jul 22 12:07:44.361973 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:44.361920 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.44:8443: connect: connection refused" Jul 22 12:07:45.667554 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:45.667511 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.45:8443: connect: connection refused" Jul 22 12:07:45.742060 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:45.742017 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="online" probeResult="failure" output="Get \"https://10.132.0.45:6567/health\": dial tcp 10.132.0.45:6567: connect: connection refused" Jul 22 12:07:51.349420 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:51.349376 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": dial tcp 10.132.0.44:6567: connect: connection refused" Jul 22 12:07:54.362026 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:54.361981 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.44:8443: connect: connection refused" Jul 22 12:07:55.667864 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:55.667816 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.45:8443: connect: connection refused" Jul 22 12:07:55.742413 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:07:55.742364 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="online" probeResult="failure" output="Get \"https://10.132.0.45:6567/health\": dial tcp 10.132.0.45:6567: connect: connection refused" Jul 22 12:08:01.348648 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:01.348607 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="online" probeResult="failure" output="Get \"https://10.132.0.44:6567/health\": dial tcp 10.132.0.44:6567: connect: connection refused" Jul 22 12:08:01.349040 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:01.348761 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:08:04.361438 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:04.361394 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.44:8443: connect: connection refused" Jul 22 12:08:04.361822 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:04.361516 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:08:05.668320 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:05.668273 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.45:8443: connect: connection refused" Jul 22 12:08:05.668753 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:05.668393 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:08:05.742125 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:05.742080 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="online" probeResult="failure" output="Get \"https://10.132.0.45:6567/health\": dial tcp 10.132.0.45:6567: connect: connection refused" Jul 22 12:08:05.742351 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:05.742274 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:08:07.165808 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.165773 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerID="051df837cab399b722f3faa797933b2bcb40a44d0f077ff79800b907cba77390" exitCode=137 Jul 22 12:08:07.165808 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.165798 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerDied","Data":"051df837cab399b722f3faa797933b2bcb40a44d0f077ff79800b907cba77390"} Jul 22 12:08:07.169493 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.169470 2579 generic.go:356] "Generic (PLEG): container finished" podID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerID="03db949a18b00a12d2164e341bdd36d78346745c19f6adfc8adf3983d9cf11f7" exitCode=137 Jul 22 12:08:07.169493 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.169490 2579 generic.go:356] "Generic (PLEG): container finished" podID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerID="d31069ba6dbbdab2409adce06808d773c07d3154bd921169c0402d3ca8ed9e02" exitCode=137 Jul 22 12:08:07.169651 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.169536 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerDied","Data":"03db949a18b00a12d2164e341bdd36d78346745c19f6adfc8adf3983d9cf11f7"} Jul 22 12:08:07.169651 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.169576 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerDied","Data":"d31069ba6dbbdab2409adce06808d773c07d3154bd921169c0402d3ca8ed9e02"} Jul 22 12:08:07.181507 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.181484 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:08:07.291615 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.291379 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47pbz\" (UniqueName: \"kubernetes.io/projected/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-kube-api-access-47pbz\") pod \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " Jul 22 12:08:07.291615 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.291491 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-online-tls\") pod \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " Jul 22 12:08:07.291615 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.291590 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-ui-tls\") pod \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " Jul 22 12:08:07.291879 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.291636 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-feast-data\") pod \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " Jul 22 12:08:07.291879 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.291684 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-offline-tls\") pod \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " Jul 22 12:08:07.291879 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.291707 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-registry-tls\") pod \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\" (UID: \"6a4a9224-4ca9-49ba-8b92-b7d2382edebd\") " Jul 22 12:08:07.292771 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.292722 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6a4a9224-4ca9-49ba-8b92-b7d2382edebd" (UID: "6a4a9224-4ca9-49ba-8b92-b7d2382edebd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:08:07.292913 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.292819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-feast-data" (OuterVolumeSpecName: "feast-data") pod "6a4a9224-4ca9-49ba-8b92-b7d2382edebd" (UID: "6a4a9224-4ca9-49ba-8b92-b7d2382edebd"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:08:07.296755 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.296721 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "6a4a9224-4ca9-49ba-8b92-b7d2382edebd" (UID: "6a4a9224-4ca9-49ba-8b92-b7d2382edebd"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:08:07.296858 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.296774 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-online-tls" (OuterVolumeSpecName: "online-tls") pod "6a4a9224-4ca9-49ba-8b92-b7d2382edebd" (UID: "6a4a9224-4ca9-49ba-8b92-b7d2382edebd"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:08:07.296858 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.296786 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-kube-api-access-47pbz" (OuterVolumeSpecName: "kube-api-access-47pbz") pod "6a4a9224-4ca9-49ba-8b92-b7d2382edebd" (UID: "6a4a9224-4ca9-49ba-8b92-b7d2382edebd"). InnerVolumeSpecName "kube-api-access-47pbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:08:07.296858 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.296818 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6a4a9224-4ca9-49ba-8b92-b7d2382edebd" (UID: "6a4a9224-4ca9-49ba-8b92-b7d2382edebd"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:08:07.332081 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.332049 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:08:07.392521 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.392483 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47pbz\" (UniqueName: \"kubernetes.io/projected/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-kube-api-access-47pbz\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.392521 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.392517 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.392739 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.392534 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-ui-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.392739 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.392543 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.392739 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.392551 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-offline-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.392739 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.392559 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/6a4a9224-4ca9-49ba-8b92-b7d2382edebd-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.492916 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.492885 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-registry-tls\") pod \"d640a04d-7b6d-4031-867b-73bb074c7caa\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " Jul 22 12:08:07.493259 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.492954 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd7cg\" (UniqueName: \"kubernetes.io/projected/d640a04d-7b6d-4031-867b-73bb074c7caa-kube-api-access-vd7cg\") pod \"d640a04d-7b6d-4031-867b-73bb074c7caa\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " Jul 22 12:08:07.493259 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.493002 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-offline-tls\") pod \"d640a04d-7b6d-4031-867b-73bb074c7caa\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " Jul 22 12:08:07.493259 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.493016 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-online-tls\") pod \"d640a04d-7b6d-4031-867b-73bb074c7caa\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " Jul 22 12:08:07.493259 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.493032 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d640a04d-7b6d-4031-867b-73bb074c7caa-feast-data\") pod \"d640a04d-7b6d-4031-867b-73bb074c7caa\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " Jul 22 12:08:07.493259 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.493065 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-ui-tls\") pod \"d640a04d-7b6d-4031-867b-73bb074c7caa\" (UID: \"d640a04d-7b6d-4031-867b-73bb074c7caa\") " Jul 22 12:08:07.493680 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.493647 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d640a04d-7b6d-4031-867b-73bb074c7caa-feast-data" (OuterVolumeSpecName: "feast-data") pod "d640a04d-7b6d-4031-867b-73bb074c7caa" (UID: "d640a04d-7b6d-4031-867b-73bb074c7caa"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:08:07.495306 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.495282 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d640a04d-7b6d-4031-867b-73bb074c7caa-kube-api-access-vd7cg" (OuterVolumeSpecName: "kube-api-access-vd7cg") pod "d640a04d-7b6d-4031-867b-73bb074c7caa" (UID: "d640a04d-7b6d-4031-867b-73bb074c7caa"). InnerVolumeSpecName "kube-api-access-vd7cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:08:07.495306 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.495293 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d640a04d-7b6d-4031-867b-73bb074c7caa" (UID: "d640a04d-7b6d-4031-867b-73bb074c7caa"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:08:07.495435 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.495306 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d640a04d-7b6d-4031-867b-73bb074c7caa" (UID: "d640a04d-7b6d-4031-867b-73bb074c7caa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:08:07.495555 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.495532 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "d640a04d-7b6d-4031-867b-73bb074c7caa" (UID: "d640a04d-7b6d-4031-867b-73bb074c7caa"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:08:07.495555 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.495549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-online-tls" (OuterVolumeSpecName: "online-tls") pod "d640a04d-7b6d-4031-867b-73bb074c7caa" (UID: "d640a04d-7b6d-4031-867b-73bb074c7caa"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:08:07.594653 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.594549 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7cg\" (UniqueName: \"kubernetes.io/projected/d640a04d-7b6d-4031-867b-73bb074c7caa-kube-api-access-vd7cg\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.594653 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.594599 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-offline-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.594653 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.594610 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-online-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.594653 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.594618 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d640a04d-7b6d-4031-867b-73bb074c7caa-feast-data\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.594653 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.594627 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-ui-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:07.594653 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:07.594649 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d640a04d-7b6d-4031-867b-73bb074c7caa-registry-tls\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:08.176885 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.176855 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" Jul 22 12:08:08.176885 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.176859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q" event={"ID":"6a4a9224-4ca9-49ba-8b92-b7d2382edebd","Type":"ContainerDied","Data":"737aa2be188ae4ce6b8c7ac9bcb7535d4b1f7bfb6901ec825ac01f7d831dda83"} Jul 22 12:08:08.177451 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.176927 2579 scope.go:117] "RemoveContainer" containerID="2c609ef74ed4b047cd677ee62d0dc69daee6c6fedef2b5dc94c43fc457f26ae1" Jul 22 12:08:08.179987 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.179956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" event={"ID":"d640a04d-7b6d-4031-867b-73bb074c7caa","Type":"ContainerDied","Data":"cfb389ea66728686fd7077ad5b278d1885ff36c29f2b8372849291a5941b1436"} Jul 22 12:08:08.180088 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.180013 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q" Jul 22 12:08:08.187275 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.187250 2579 scope.go:117] "RemoveContainer" containerID="051df837cab399b722f3faa797933b2bcb40a44d0f077ff79800b907cba77390" Jul 22 12:08:08.195275 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.195258 2579 scope.go:117] "RemoveContainer" containerID="9926a9891625ebf199c89aa77846c003670907640cfc87f5971c8c4dbf88dba3" Jul 22 12:08:08.204078 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.204055 2579 scope.go:117] "RemoveContainer" containerID="3eeeb7f209d6fdccb45bd768c0d1d79c3ae0d3ef34266c5fac3a862d20ed274e" Jul 22 12:08:08.206310 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.206282 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q"] Jul 22 12:08:08.206956 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.206930 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rnz2q"] Jul 22 12:08:08.212312 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.212288 2579 scope.go:117] "RemoveContainer" containerID="fb2700b499d09ce1c309a10408b63b3c531d7319377af707f1016003f3554978" Jul 22 12:08:08.218450 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.218428 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q"] Jul 22 12:08:08.220076 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.220063 2579 scope.go:117] "RemoveContainer" containerID="a3a21a17a85af9d33ce973a5039f27baf956a59069c734a006727ba08c927518" Jul 22 12:08:08.222835 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.222808 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-c8w6q"] Jul 22 12:08:08.228058 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.228037 2579 scope.go:117] "RemoveContainer" containerID="03db949a18b00a12d2164e341bdd36d78346745c19f6adfc8adf3983d9cf11f7" Jul 22 12:08:08.235517 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.235497 2579 scope.go:117] "RemoveContainer" containerID="15a93f120088e951837ecd594927fd3680f02bbeb2a6b4be04e4e8dbea6dc23d" Jul 22 12:08:08.243072 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.243055 2579 scope.go:117] "RemoveContainer" containerID="d31069ba6dbbdab2409adce06808d773c07d3154bd921169c0402d3ca8ed9e02" Jul 22 12:08:08.250635 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.250618 2579 scope.go:117] "RemoveContainer" containerID="bba1b85f40f12ba7dd191aae95fe0396b00f9e028618a21d8a0aa0ceac56fd04" Jul 22 12:08:08.257741 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:08.257722 2579 scope.go:117] "RemoveContainer" containerID="2a7256eb995c260df4de7cf2619d3c3279c9d1c25fda91a1ab9f8499ee44de3f" Jul 22 12:08:10.206850 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:10.206816 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" path="/var/lib/kubelet/pods/6a4a9224-4ca9-49ba-8b92-b7d2382edebd/volumes" Jul 22 12:08:10.207417 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:10.207402 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" path="/var/lib/kubelet/pods/d640a04d-7b6d-4031-867b-73bb074c7caa/volumes" Jul 22 12:08:26.161952 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.161910 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c5mtd/must-gather-hqq2t"] Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162273 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="offline" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162286 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="offline" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162300 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="online" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162308 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="online" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162322 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="feast-apply" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162329 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="feast-apply" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162335 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="offline" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162340 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="offline" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162347 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="ui" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162352 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="ui" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162358 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="feast-init" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162363 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="feast-init" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162372 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="feast-init" Jul 22 12:08:26.162375 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162377 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="feast-init" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162390 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="feast-apply" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162398 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="feast-apply" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162404 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="online" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162409 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="online" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162414 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="registry" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162419 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="registry" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162424 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="ui" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162429 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="ui" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162494 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="offline" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162503 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="registry" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162510 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="online" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162515 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="ui" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162521 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="online" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162528 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a4a9224-4ca9-49ba-8b92-b7d2382edebd" containerName="ui" Jul 22 12:08:26.162828 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.162534 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d640a04d-7b6d-4031-867b-73bb074c7caa" containerName="offline" Jul 22 12:08:26.166967 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.166947 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:26.170042 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.170024 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c5mtd\"/\"default-dockercfg-ncv2v\"" Jul 22 12:08:26.171398 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.171374 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c5mtd\"/\"openshift-service-ca.crt\"" Jul 22 12:08:26.171469 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.171453 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c5mtd\"/\"kube-root-ca.crt\"" Jul 22 12:08:26.180102 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.180081 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c5mtd/must-gather-hqq2t"] Jul 22 12:08:26.258146 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.258109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2lqs\" (UniqueName: \"kubernetes.io/projected/0b8bce64-6286-49f6-97eb-78e21b40d386-kube-api-access-x2lqs\") pod \"must-gather-hqq2t\" (UID: \"0b8bce64-6286-49f6-97eb-78e21b40d386\") " pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:26.258372 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.258163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b8bce64-6286-49f6-97eb-78e21b40d386-must-gather-output\") pod \"must-gather-hqq2t\" (UID: \"0b8bce64-6286-49f6-97eb-78e21b40d386\") " pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:26.359450 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.359406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b8bce64-6286-49f6-97eb-78e21b40d386-must-gather-output\") pod \"must-gather-hqq2t\" (UID: \"0b8bce64-6286-49f6-97eb-78e21b40d386\") " pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:26.359662 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.359506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2lqs\" (UniqueName: \"kubernetes.io/projected/0b8bce64-6286-49f6-97eb-78e21b40d386-kube-api-access-x2lqs\") pod \"must-gather-hqq2t\" (UID: \"0b8bce64-6286-49f6-97eb-78e21b40d386\") " pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:26.359769 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.359743 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b8bce64-6286-49f6-97eb-78e21b40d386-must-gather-output\") pod \"must-gather-hqq2t\" (UID: \"0b8bce64-6286-49f6-97eb-78e21b40d386\") " pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:26.368945 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.368917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2lqs\" (UniqueName: \"kubernetes.io/projected/0b8bce64-6286-49f6-97eb-78e21b40d386-kube-api-access-x2lqs\") pod \"must-gather-hqq2t\" (UID: \"0b8bce64-6286-49f6-97eb-78e21b40d386\") " pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:26.484581 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.484497 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:26.611481 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.611452 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c5mtd/must-gather-hqq2t"] Jul 22 12:08:26.613766 ip-10-0-132-210 kubenswrapper[2579]: W0722 12:08:26.613741 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b8bce64_6286_49f6_97eb_78e21b40d386.slice/crio-f06f3f0565573181af22991684d3a26ab7ae1f9b8672388fbd1f338bf10227a6 WatchSource:0}: Error finding container f06f3f0565573181af22991684d3a26ab7ae1f9b8672388fbd1f338bf10227a6: Status 404 returned error can't find the container with id f06f3f0565573181af22991684d3a26ab7ae1f9b8672388fbd1f338bf10227a6 Jul 22 12:08:26.615985 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:26.615962 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:08:27.254356 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:27.254315 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" event={"ID":"0b8bce64-6286-49f6-97eb-78e21b40d386","Type":"ContainerStarted","Data":"f06f3f0565573181af22991684d3a26ab7ae1f9b8672388fbd1f338bf10227a6"} Jul 22 12:08:33.287796 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:33.287750 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" event={"ID":"0b8bce64-6286-49f6-97eb-78e21b40d386","Type":"ContainerStarted","Data":"59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b"} Jul 22 12:08:33.287796 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:33.287806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" event={"ID":"0b8bce64-6286-49f6-97eb-78e21b40d386","Type":"ContainerStarted","Data":"b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30"} Jul 22 12:08:33.320547 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:33.320477 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" podStartSLOduration=1.076801437 podStartE2EDuration="7.320451487s" totalImagesPullingTime="6.24365005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:08:33.316479282 +0000 UTC m=+1949.615454239" watchObservedRunningTime="2026-07-22 12:08:33.320451487 +0000 UTC m=+1949.619426447" Jul 22 12:08:43.336142 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:43.336102 2579 generic.go:356] "Generic (PLEG): container finished" podID="0b8bce64-6286-49f6-97eb-78e21b40d386" containerID="b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30" exitCode=0 Jul 22 12:08:43.336626 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:43.336173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" event={"ID":"0b8bce64-6286-49f6-97eb-78e21b40d386","Type":"ContainerDied","Data":"b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30"} Jul 22 12:08:43.336626 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:43.336545 2579 scope.go:117] "RemoveContainer" containerID="b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30" Jul 22 12:08:43.671862 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:43.671829 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c5mtd_must-gather-hqq2t_0b8bce64-6286-49f6-97eb-78e21b40d386/gather/0.log" Jul 22 12:08:46.844819 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:46.844788 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gcrmm_1aca7ae8-e7fc-417d-840b-a7fa1155723f/global-pull-secret-syncer/0.log" Jul 22 12:08:47.046281 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:47.046250 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4vndv_1df4d1a6-bc29-4396-8ed8-a0e40c91c0f0/konnectivity-agent/0.log" Jul 22 12:08:47.128128 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:47.128033 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-210.ec2.internal_6115ca8597103d232ce4b4815a5dee79/haproxy/0.log" Jul 22 12:08:49.055520 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.055481 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-c5mtd/must-gather-hqq2t"] Jul 22 12:08:49.056000 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.055683 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" containerName="copy" containerID="cri-o://59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b" gracePeriod=2 Jul 22 12:08:49.064093 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.064069 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-c5mtd/must-gather-hqq2t"] Jul 22 12:08:49.289388 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.289361 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c5mtd_must-gather-hqq2t_0b8bce64-6286-49f6-97eb-78e21b40d386/copy/0.log" Jul 22 12:08:49.290879 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.290861 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:49.293387 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:08:49.293362 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hqq2t\" is forbidden: User \"system:node:ip-10-0-132-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c5mtd\": no relationship found between node 'ip-10-0-132-210.ec2.internal' and this object" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:49.361726 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.361659 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c5mtd_must-gather-hqq2t_0b8bce64-6286-49f6-97eb-78e21b40d386/copy/0.log" Jul 22 12:08:49.361726 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.361687 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b8bce64-6286-49f6-97eb-78e21b40d386-must-gather-output\") pod \"0b8bce64-6286-49f6-97eb-78e21b40d386\" (UID: \"0b8bce64-6286-49f6-97eb-78e21b40d386\") " Jul 22 12:08:49.361923 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.361741 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2lqs\" (UniqueName: \"kubernetes.io/projected/0b8bce64-6286-49f6-97eb-78e21b40d386-kube-api-access-x2lqs\") pod \"0b8bce64-6286-49f6-97eb-78e21b40d386\" (UID: \"0b8bce64-6286-49f6-97eb-78e21b40d386\") " Jul 22 12:08:49.362073 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.362041 2579 generic.go:356] "Generic (PLEG): container finished" podID="0b8bce64-6286-49f6-97eb-78e21b40d386" containerID="59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b" exitCode=143 Jul 22 12:08:49.362183 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.362073 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8bce64-6286-49f6-97eb-78e21b40d386-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0b8bce64-6286-49f6-97eb-78e21b40d386" (UID: "0b8bce64-6286-49f6-97eb-78e21b40d386"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:08:49.362183 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.362086 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:49.362183 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.362100 2579 scope.go:117] "RemoveContainer" containerID="59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b" Jul 22 12:08:49.364036 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.364013 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8bce64-6286-49f6-97eb-78e21b40d386-kube-api-access-x2lqs" (OuterVolumeSpecName: "kube-api-access-x2lqs") pod "0b8bce64-6286-49f6-97eb-78e21b40d386" (UID: "0b8bce64-6286-49f6-97eb-78e21b40d386"). InnerVolumeSpecName "kube-api-access-x2lqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:08:49.364991 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:08:49.364966 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hqq2t\" is forbidden: User \"system:node:ip-10-0-132-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c5mtd\": no relationship found between node 'ip-10-0-132-210.ec2.internal' and this object" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:49.373521 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.373505 2579 scope.go:117] "RemoveContainer" containerID="b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30" Jul 22 12:08:49.385478 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.385462 2579 scope.go:117] "RemoveContainer" containerID="59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b" Jul 22 12:08:49.385735 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:08:49.385714 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b\": container with ID starting with 59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b not found: ID does not exist" containerID="59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b" Jul 22 12:08:49.385801 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.385743 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b"} err="failed to get container status \"59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b\": rpc error: code = NotFound desc = could not find container \"59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b\": container with ID starting with 59885e7c70646ddb5591af1ed23d7c3c3f2f1ec8ecfade5f79f93fd03bb9b71b not found: ID does not exist" Jul 22 12:08:49.385801 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.385760 2579 scope.go:117] "RemoveContainer" containerID="b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30" Jul 22 12:08:49.385996 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:08:49.385979 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30\": container with ID starting with b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30 not found: ID does not exist" containerID="b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30" Jul 22 12:08:49.386040 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.386002 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30"} err="failed to get container status \"b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30\": rpc error: code = NotFound desc = could not find container \"b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30\": container with ID starting with b1b7572b3bd941dc0851a7f80a9b986ada32ca938ceda89ca742e72cb0b72a30 not found: ID does not exist" Jul 22 12:08:49.463004 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.462973 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2lqs\" (UniqueName: \"kubernetes.io/projected/0b8bce64-6286-49f6-97eb-78e21b40d386-kube-api-access-x2lqs\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:49.463004 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:49.463000 2579 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b8bce64-6286-49f6-97eb-78e21b40d386-must-gather-output\") on node \"ip-10-0-132-210.ec2.internal\" DevicePath \"\"" Jul 22 12:08:49.672865 ip-10-0-132-210 kubenswrapper[2579]: E0722 12:08:49.672834 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hqq2t\" is forbidden: User \"system:node:ip-10-0-132-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c5mtd\": no relationship found between node 'ip-10-0-132-210.ec2.internal' and this object" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" pod="openshift-must-gather-c5mtd/must-gather-hqq2t" Jul 22 12:08:50.206664 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:50.206637 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" path="/var/lib/kubelet/pods/0b8bce64-6286-49f6-97eb-78e21b40d386/volumes" Jul 22 12:08:50.975708 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:50.975673 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-b4nc8_96025fea-bf2d-43e3-9506-7ebea486627c/cluster-monitoring-operator/0.log" Jul 22 12:08:51.006657 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.006631 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-p6ght_a8758964-6f84-4261-b4f4-19c44ea2e7c0/kube-state-metrics/0.log" Jul 22 12:08:51.029303 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.029281 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-p6ght_a8758964-6f84-4261-b4f4-19c44ea2e7c0/kube-rbac-proxy-main/0.log" Jul 22 12:08:51.054321 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.054299 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-p6ght_a8758964-6f84-4261-b4f4-19c44ea2e7c0/kube-rbac-proxy-self/0.log" Jul 22 12:08:51.079563 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.079542 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-676878bd5-xlljw_997ce5d8-3759-4546-a771-72ccce27cd29/metrics-server/0.log" Jul 22 12:08:51.307594 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.307513 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh88r_a86e44d1-3d0d-4e6f-947c-17736beed5dc/node-exporter/0.log" Jul 22 12:08:51.328250 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.328223 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh88r_a86e44d1-3d0d-4e6f-947c-17736beed5dc/kube-rbac-proxy/0.log" Jul 22 12:08:51.349493 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.349470 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh88r_a86e44d1-3d0d-4e6f-947c-17736beed5dc/init-textfile/0.log" Jul 22 12:08:51.379440 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.379416 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-m4s4g_8dd28007-79e6-42a6-8794-40abf5fa6a21/kube-rbac-proxy-main/0.log" Jul 22 12:08:51.405111 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.405089 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-m4s4g_8dd28007-79e6-42a6-8794-40abf5fa6a21/kube-rbac-proxy-self/0.log" Jul 22 12:08:51.430965 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.430936 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-m4s4g_8dd28007-79e6-42a6-8794-40abf5fa6a21/openshift-state-metrics/0.log" Jul 22 12:08:51.694246 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.694220 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-srtkc_7fc81cfe-5353-4f61-a55c-e096a5b1d34f/prometheus-operator-admission-webhook/0.log" Jul 22 12:08:51.837456 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.837426 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dc67b8c86-4rngp_6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c/thanos-query/0.log" Jul 22 12:08:51.865750 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.865722 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dc67b8c86-4rngp_6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c/kube-rbac-proxy-web/0.log" Jul 22 12:08:51.905615 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.905585 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dc67b8c86-4rngp_6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c/kube-rbac-proxy/0.log" Jul 22 12:08:51.934845 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.934818 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dc67b8c86-4rngp_6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c/prom-label-proxy/0.log" Jul 22 12:08:51.956815 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.956745 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dc67b8c86-4rngp_6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c/kube-rbac-proxy-rules/0.log" Jul 22 12:08:51.984647 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:51.984617 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dc67b8c86-4rngp_6f6f48e3-cad1-4e36-8c64-e2fe10cb9e9c/kube-rbac-proxy-metrics/0.log" Jul 22 12:08:53.625712 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.625677 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/1.log" Jul 22 12:08:53.630338 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.630306 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-t5smg_a4ca318a-3b21-42da-9525-6e907b40970d/console-operator/2.log" Jul 22 12:08:53.909149 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.909115 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt"] Jul 22 12:08:53.909445 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.909432 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" containerName="copy" Jul 22 12:08:53.909445 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.909446 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" containerName="copy" Jul 22 12:08:53.909524 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.909460 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" containerName="gather" Jul 22 12:08:53.909524 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.909466 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" containerName="gather" Jul 22 12:08:53.909595 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.909529 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" containerName="gather" Jul 22 12:08:53.909595 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.909538 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8bce64-6286-49f6-97eb-78e21b40d386" containerName="copy" Jul 22 12:08:53.913124 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.913102 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:53.915603 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.915580 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x7fzg\"/\"kube-root-ca.crt\"" Jul 22 12:08:53.915839 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.915821 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x7fzg\"/\"openshift-service-ca.crt\"" Jul 22 12:08:53.916689 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.916673 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x7fzg\"/\"default-dockercfg-q5c2q\"" Jul 22 12:08:53.922283 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:53.922252 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt"] Jul 22 12:08:54.001354 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.001310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-sys\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.001354 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.001357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-podres\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.001589 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.001453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-proc\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.001589 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.001497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsms9\" (UniqueName: \"kubernetes.io/projected/e090948e-f728-4588-82f8-110acbb1dd5e-kube-api-access-hsms9\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.001589 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.001542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-lib-modules\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.070551 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.070521 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b9c7758c4-4mp8m_21f2fdfc-9b45-49cb-bdb5-860ddbb362a2/console/0.log" Jul 22 12:08:54.102247 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.102188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-lib-modules\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.102416 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.102269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-sys\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.102416 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.102299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-podres\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.102416 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.102379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-proc\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.102416 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.102380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-lib-modules\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.102416 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.102381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-sys\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.102651 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.102433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-proc\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.102651 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.102452 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e090948e-f728-4588-82f8-110acbb1dd5e-podres\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.102651 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.102470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsms9\" (UniqueName: \"kubernetes.io/projected/e090948e-f728-4588-82f8-110acbb1dd5e-kube-api-access-hsms9\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.116185 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.116166 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsms9\" (UniqueName: \"kubernetes.io/projected/e090948e-f728-4588-82f8-110acbb1dd5e-kube-api-access-hsms9\") pod \"perf-node-gather-daemonset-nvspt\" (UID: \"e090948e-f728-4588-82f8-110acbb1dd5e\") " pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.223586 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.223513 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:54.354147 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.354116 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt"] Jul 22 12:08:54.357892 ip-10-0-132-210 kubenswrapper[2579]: W0722 12:08:54.357861 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode090948e_f728_4588_82f8_110acbb1dd5e.slice/crio-3e9ad0b1e3ee3b1513a0457eabf8a85757d16c74dc5a6d30119f5a19b1651a9c WatchSource:0}: Error finding container 3e9ad0b1e3ee3b1513a0457eabf8a85757d16c74dc5a6d30119f5a19b1651a9c: Status 404 returned error can't find the container with id 3e9ad0b1e3ee3b1513a0457eabf8a85757d16c74dc5a6d30119f5a19b1651a9c Jul 22 12:08:54.384092 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.384067 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" event={"ID":"e090948e-f728-4588-82f8-110acbb1dd5e","Type":"ContainerStarted","Data":"3e9ad0b1e3ee3b1513a0457eabf8a85757d16c74dc5a6d30119f5a19b1651a9c"} Jul 22 12:08:54.638718 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:54.638690 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-xtxws_7a58d7a4-7171-4103-8e5f-1d921c7572af/volume-data-source-validator/0.log" Jul 22 12:08:55.389519 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:55.389485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" event={"ID":"e090948e-f728-4588-82f8-110acbb1dd5e","Type":"ContainerStarted","Data":"53f6c4c9ed406fc9db073686bd565d6c17ee791f39de1c34a61f1e28c3eac2d0"} Jul 22 12:08:55.389683 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:55.389554 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:08:55.408638 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:55.408586 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" podStartSLOduration=2.408571249 podStartE2EDuration="2.408571249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:08:55.408180213 +0000 UTC m=+1971.707155196" watchObservedRunningTime="2026-07-22 12:08:55.408571249 +0000 UTC m=+1971.707546207" Jul 22 12:08:55.461974 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:55.461922 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cp489_869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2/dns/0.log" Jul 22 12:08:55.483218 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:55.483171 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cp489_869c1d9a-ea5f-4ab4-a3f9-9c9a862985c2/kube-rbac-proxy/0.log" Jul 22 12:08:55.579942 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:55.579914 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ndr58_3d93eebd-7f1a-48fd-a0e3-51fe4ab46250/dns-node-resolver/0.log" Jul 22 12:08:56.053776 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:56.053743 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mwg4w_11037cad-c16a-4b04-b5ee-1287f5caef55/node-ca/0.log" Jul 22 12:08:56.714206 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:56.714095 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67fc875744-bt6br_1c5fe7bc-16c8-44d7-b77a-8929bc72ca26/router/0.log" Jul 22 12:08:57.027531 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:57.027428 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r45q9_7879951c-c75e-42ed-a411-5cebada8a304/serve-healthcheck-canary/0.log" Jul 22 12:08:57.396800 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:57.396758 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-kzsvj_e0d0b446-3750-4f25-b181-87b1b351863d/insights-operator/0.log" Jul 22 12:08:57.397605 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:57.397584 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-kzsvj_e0d0b446-3750-4f25-b181-87b1b351863d/insights-operator/1.log" Jul 22 12:08:57.542076 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:57.542039 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v4wgf_386b36f3-f8d6-4928-a089-79e0b97db4f4/kube-rbac-proxy/0.log" Jul 22 12:08:57.560151 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:57.560125 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v4wgf_386b36f3-f8d6-4928-a089-79e0b97db4f4/exporter/0.log" Jul 22 12:08:57.581048 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:08:57.581024 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v4wgf_386b36f3-f8d6-4928-a089-79e0b97db4f4/extractor/0.log" Jul 22 12:09:01.403612 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:01.403580 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x7fzg/perf-node-gather-daemonset-nvspt" Jul 22 12:09:02.091030 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:02.090994 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-j5bf6_41df392a-72e8-4b96-b6dd-a8eaf50b322b/kube-storage-version-migrator-operator/1.log" Jul 22 12:09:02.091967 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:02.091948 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-j5bf6_41df392a-72e8-4b96-b6dd-a8eaf50b322b/kube-storage-version-migrator-operator/0.log" Jul 22 12:09:03.158821 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:03.158789 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbflp_e9412ed9-53f3-4469-99aa-119dab079331/kube-multus-additional-cni-plugins/0.log" Jul 22 12:09:03.182061 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:03.182029 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbflp_e9412ed9-53f3-4469-99aa-119dab079331/egress-router-binary-copy/0.log" Jul 22 12:09:03.203448 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:03.203414 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbflp_e9412ed9-53f3-4469-99aa-119dab079331/cni-plugins/0.log" Jul 22 12:09:03.227346 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:03.227318 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbflp_e9412ed9-53f3-4469-99aa-119dab079331/bond-cni-plugin/0.log" Jul 22 12:09:03.249058 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:03.248961 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbflp_e9412ed9-53f3-4469-99aa-119dab079331/routeoverride-cni/0.log" Jul 22 12:09:03.270517 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:03.270485 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbflp_e9412ed9-53f3-4469-99aa-119dab079331/whereabouts-cni-bincopy/0.log" Jul 22 12:09:03.289778 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:03.289743 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbflp_e9412ed9-53f3-4469-99aa-119dab079331/whereabouts-cni/0.log" Jul 22 12:09:03.821689 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:03.821638 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lmkbc_787909e2-590a-44e5-baa0-d5d7b711aa6b/kube-multus/0.log" Jul 22 12:09:03.943909 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:03.943882 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x92mg_64781c58-80c2-4b43-bc93-8bd1a7df80ca/network-metrics-daemon/0.log" Jul 22 12:09:03.967179 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:03.967143 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x92mg_64781c58-80c2-4b43-bc93-8bd1a7df80ca/kube-rbac-proxy/0.log" Jul 22 12:09:04.732728 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:04.732679 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sctd_51efd00e-bce7-47b2-92ee-94fcd52ae3ab/ovn-controller/0.log" Jul 22 12:09:04.759182 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:04.759156 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sctd_51efd00e-bce7-47b2-92ee-94fcd52ae3ab/ovn-acl-logging/0.log" Jul 22 12:09:04.788175 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:04.788151 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sctd_51efd00e-bce7-47b2-92ee-94fcd52ae3ab/kube-rbac-proxy-node/0.log" Jul 22 12:09:04.814623 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:04.814584 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sctd_51efd00e-bce7-47b2-92ee-94fcd52ae3ab/kube-rbac-proxy-ovn-metrics/0.log" Jul 22 12:09:04.836568 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:04.836541 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sctd_51efd00e-bce7-47b2-92ee-94fcd52ae3ab/northd/0.log" Jul 22 12:09:04.863395 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:04.863370 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sctd_51efd00e-bce7-47b2-92ee-94fcd52ae3ab/nbdb/0.log" Jul 22 12:09:04.896665 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:04.896642 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sctd_51efd00e-bce7-47b2-92ee-94fcd52ae3ab/sbdb/0.log" Jul 22 12:09:04.951312 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:04.951279 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sctd_51efd00e-bce7-47b2-92ee-94fcd52ae3ab/ovnkube-controller/0.log" Jul 22 12:09:06.442868 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:06.442828 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7bc5b6bf85-r64wt_62aac9b7-2b61-47c3-b68f-0cde3dee4534/check-endpoints/0.log" Jul 22 12:09:06.514334 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:06.514298 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-j7pfq_60af647a-0dc7-4414-8cac-a61393b4cb51/network-check-target-container/0.log" Jul 22 12:09:07.392638 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:07.392609 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gf2wn_3b0b3fa4-bc82-4499-a622-3fb464453a1c/iptables-alerter/0.log" Jul 22 12:09:08.104911 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:08.104882 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-42rnz_df03ccdb-d98c-4df0-884a-d674e4bd3dcd/tuned/0.log" Jul 22 12:09:10.101471 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:10.101440 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-79b7c55f87-bqc7c_e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa/cluster-samples-operator/0.log" Jul 22 12:09:10.127377 ip-10-0-132-210 kubenswrapper[2579]: I0722 12:09:10.127351 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-79b7c55f87-bqc7c_e7fdcdb9-37de-45e1-96c2-ef723ae4bbaa/cluster-samples-operator-watch/0.log"