Jul 29 08:15:56.309085 ip-10-0-131-90 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 08:15:56.309101 ip-10-0-131-90 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 08:15:56.309111 ip-10-0-131-90 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 08:15:56.309470 ip-10-0-131-90 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 08:16:06.259712 ip-10-0-131-90 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 29 08:16:06.259725 ip-10-0-131-90 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c4a1636e5b154642bbd7b4853d385fba -- Jul 29 08:17:59.568946 ip-10-0-131-90 systemd[1]: Starting Kubernetes Kubelet... Jul 29 08:17:59.988954 ip-10-0-131-90 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 08:17:59.988954 ip-10-0-131-90 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 08:17:59.988954 ip-10-0-131-90 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 08:17:59.988954 ip-10-0-131-90 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 08:17:59.989614 ip-10-0-131-90 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 08:17:59.989614 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.989030 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 08:17:59.993517 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993497 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:17:59.993517 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993516 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993520 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993523 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993526 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993529 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993532 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993535 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993539 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993541 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993544 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993547 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993549 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993552 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993554 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993562 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993564 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993567 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993570 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993572 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993575 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:17:59.993585 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993577 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993580 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993582 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993585 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993588 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993590 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993594 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993597 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993601 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993605 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993609 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993612 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993629 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993632 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993635 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993637 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993640 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993642 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993644 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:17:59.994071 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993647 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993650 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993652 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993655 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993657 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993660 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993662 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993664 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993667 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993670 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993672 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993674 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993677 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993679 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993681 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993685 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993688 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993690 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993693 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993695 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:17:59.994534 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993698 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993700 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993702 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993706 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993710 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993712 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993715 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993717 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993720 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993723 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993725 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993728 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993731 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993735 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993737 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993739 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993742 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993744 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993746 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993749 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:17:59.995034 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993751 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993754 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993756 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993760 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993763 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993766 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993769 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993773 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993776 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993780 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993782 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993785 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993788 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993791 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993793 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993796 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993799 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993801 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.993804 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994491 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:17:59.995508 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994497 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994500 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994503 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994506 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994509 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994512 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994514 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994517 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994519 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994522 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994524 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994527 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994529 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994532 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994534 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994537 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994539 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994542 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994544 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:17:59.996055 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994548 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994551 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994554 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994556 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994560 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994563 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994566 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994568 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994571 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994573 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994576 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994579 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994581 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994583 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994586 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994589 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994591 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994593 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994596 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994599 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:17:59.996523 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994601 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994604 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994606 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994608 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994610 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994613 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994634 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994639 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994643 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994647 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994651 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994654 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994657 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994660 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994663 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994665 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994670 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994672 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994675 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994678 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:17:59.997016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994680 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994683 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994685 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994688 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994691 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994693 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994696 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994700 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994703 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994708 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994711 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994714 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994717 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994720 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994722 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994725 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994727 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994729 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994732 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:17:59.997507 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994734 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994738 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994741 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994744 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994747 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994751 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994753 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994756 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994758 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994761 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994763 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994766 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994768 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994771 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994774 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994776 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994778 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994781 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994783 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.994786 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:17:59.997970 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995171 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995176 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995179 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995182 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995185 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995188 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995192 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995195 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995198 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995201 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995204 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995206 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995209 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995211 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995213 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995216 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995220 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995224 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995227 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:17:59.998452 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995230 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995233 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995236 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995238 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995241 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995243 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995246 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995248 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995251 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995253 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995255 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995257 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995260 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995263 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995265 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995268 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995270 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995273 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995278 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995281 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:17:59.998938 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995283 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995286 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995289 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995291 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995294 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995296 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995298 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995301 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995304 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995308 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995311 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995313 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995316 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995319 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995321 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995324 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995326 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995329 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995331 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:17:59.999500 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995334 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995336 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995338 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995341 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995343 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995345 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995348 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995351 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995354 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995356 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995358 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995361 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995364 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995366 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995368 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995371 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995373 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995376 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995379 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995381 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:18:00.000001 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995383 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995386 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995388 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995390 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995393 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995395 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995398 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995400 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995402 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995404 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995407 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995409 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995411 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995414 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995416 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995419 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995421 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995423 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995426 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995429 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:18:00.000482 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.995431 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995525 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995537 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995545 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995550 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995555 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995558 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995563 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995567 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995570 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995573 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995576 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995579 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995582 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995587 2582 flags.go:64] FLAG: --cgroup-root="" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995590 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995593 2582 flags.go:64] FLAG: --client-ca-file="" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995596 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995599 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995604 2582 flags.go:64] FLAG: --cluster-domain="" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995607 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995610 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995613 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995632 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 08:18:00.001009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995636 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995639 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995642 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995645 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995648 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995651 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995654 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995658 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995660 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995665 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995668 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995670 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995673 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995676 2582 flags.go:64] FLAG: --enable-server="true" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995680 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995684 2582 flags.go:64] FLAG: --event-burst="100" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995687 2582 flags.go:64] FLAG: --event-qps="50" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995690 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995693 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995695 2582 flags.go:64] FLAG: --eviction-hard="" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995699 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995702 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995706 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995709 2582 flags.go:64] FLAG: --eviction-soft="" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995712 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 08:18:00.001581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995714 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995717 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995720 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995723 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995726 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995729 2582 flags.go:64] FLAG: --feature-gates="" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995733 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995735 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995738 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995742 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995754 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995757 2582 flags.go:64] FLAG: --help="false" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995760 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995764 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995768 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995770 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995774 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995777 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995780 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995782 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995785 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995788 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995791 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995795 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 08:18:00.002213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995798 2582 flags.go:64] FLAG: --kube-reserved="" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995801 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995803 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995806 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995809 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995813 2582 flags.go:64] FLAG: --lock-file="" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995815 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995818 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995821 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995827 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995830 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995833 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995836 2582 flags.go:64] FLAG: --logging-format="text" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995839 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995842 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995844 2582 flags.go:64] FLAG: --manifest-url="" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995847 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995852 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995855 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995859 2582 flags.go:64] FLAG: --max-pods="110" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995862 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995866 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995870 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995873 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995876 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 08:18:00.002824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995879 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995882 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995889 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995892 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995895 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995898 2582 flags.go:64] FLAG: --pod-cidr="" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995901 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995907 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995910 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995913 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995916 2582 flags.go:64] FLAG: --port="10250" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995919 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995922 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06e7d11fcc3e0d514" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995926 2582 flags.go:64] FLAG: --qos-reserved="" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995929 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995932 2582 flags.go:64] FLAG: --register-node="true" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995934 2582 flags.go:64] FLAG: --register-with-taints="" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995938 2582 flags.go:64] FLAG: --registry-burst="10" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995941 2582 flags.go:64] FLAG: --registry-qps="5" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995944 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995947 2582 flags.go:64] FLAG: --reserved-memory="" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995950 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995953 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995956 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995959 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 08:18:00.003430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995961 2582 flags.go:64] FLAG: --runonce="false" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995964 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995967 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995971 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995973 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995976 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995979 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995982 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995985 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995988 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995991 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995993 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995996 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.995999 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996002 2582 flags.go:64] FLAG: --system-cgroups="" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996005 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996010 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996012 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996015 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996019 2582 flags.go:64] FLAG: --tls-min-version="" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996023 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996026 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996028 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996031 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996034 2582 flags.go:64] FLAG: --v="2" Jul 29 08:18:00.004052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996038 2582 flags.go:64] FLAG: --version="false" Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996042 2582 flags.go:64] FLAG: --vmodule="" Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996046 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996049 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996170 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996173 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996176 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996179 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996183 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996187 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996191 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996194 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996196 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996199 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996202 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996204 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996207 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996209 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996212 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996214 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:18:00.004817 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996217 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996219 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996224 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996227 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996231 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996233 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996236 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996241 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996243 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996246 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996249 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996252 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996254 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996257 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996260 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996262 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996265 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996268 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996270 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:18:00.005316 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996272 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996275 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996278 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996282 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996285 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996287 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996289 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996292 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996294 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996296 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996299 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996301 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996304 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996306 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996308 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996311 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996314 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996316 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996319 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996321 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:18:00.005812 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996325 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996327 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996330 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996332 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996335 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996337 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996340 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996342 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996345 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996347 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996350 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996352 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996354 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996357 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996359 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996363 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996366 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996369 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996371 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996373 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:18:00.006303 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996376 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996378 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996380 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996383 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996385 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996388 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996390 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996393 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996395 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996398 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996401 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996403 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996407 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996410 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996413 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996415 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996418 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996420 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996422 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996425 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:18:00.006806 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996428 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996430 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996434 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:17:59.996437 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:17:59.996443 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.003915 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.003932 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.005790 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.005803 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005866 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005871 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005875 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005878 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005881 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:18:00.007294 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005884 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005886 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005889 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005892 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005895 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005897 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005900 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005904 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005906 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005909 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005911 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005914 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005917 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005919 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005922 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005924 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005926 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005929 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005931 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:18:00.007660 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005934 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005938 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005941 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005944 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005948 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005951 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005954 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005974 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005978 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005981 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005985 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005988 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005991 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005993 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005996 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.005998 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006001 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006004 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006008 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:18:00.008130 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006011 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006013 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006016 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006019 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006021 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006024 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006026 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006029 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006031 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006035 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006038 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006040 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006044 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006047 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006049 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006052 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006054 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006058 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006061 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:18:00.008598 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006063 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006066 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006068 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006071 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006073 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006076 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006078 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006081 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006084 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006086 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006089 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006091 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006094 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006096 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006100 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006102 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006105 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006108 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006110 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006113 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:18:00.009084 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006116 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006119 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006121 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006123 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006126 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006128 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006130 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006133 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006135 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006138 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006140 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006143 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006146 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006148 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006150 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006153 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:18:00.009565 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006155 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.006160 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006279 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006284 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006288 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006290 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006293 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006295 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006297 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006300 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006302 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006305 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006308 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006311 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006313 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:18:00.010008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006317 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006319 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006321 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006324 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006326 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006329 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006331 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006334 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006336 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006338 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006341 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006343 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006345 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006348 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006351 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006353 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006356 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006359 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006361 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:18:00.010380 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006363 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006366 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006368 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006371 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006373 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006375 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006378 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006381 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006385 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006388 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006390 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006393 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006396 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006399 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006401 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006403 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006406 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006408 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006410 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:18:00.010909 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006414 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006418 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006421 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006423 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006426 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006428 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006431 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006434 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006436 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006438 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006441 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006443 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006446 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006448 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006450 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006453 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006455 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006457 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006460 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006462 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:18:00.011357 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006465 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006467 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006470 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006472 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006475 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006477 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006481 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006485 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006487 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006490 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006493 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006495 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006498 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006501 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006503 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006505 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006508 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006510 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006512 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006515 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:18:00.011856 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006518 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006520 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006523 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006525 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006527 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006530 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006532 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:00.006535 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.006539 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.006685 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.008804 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.010337 2582 server.go:1019] "Starting client certificate rotation" Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.010443 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 08:18:00.012352 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.010505 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 08:18:00.013184 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.013170 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 08:18:00.016572 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.016548 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 08:18:00.029998 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.029975 2582 log.go:25] "Validated CRI v1 runtime API" Jul 29 08:18:00.032134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.032118 2582 log.go:25] "Validated CRI v1 image API" Jul 29 08:18:00.032952 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.032938 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 08:18:00.035588 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.035567 2582 fs.go:135] Filesystem UUIDs: map[12b2caa7-0e89-4d2c-96cc-a9ceec933478:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ddc7ea23-965c-45fe-8953-e3612aef7a53:/dev/nvme0n1p3] Jul 29 08:18:00.035672 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.035589 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 08:18:00.040580 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.040451 2582 manager.go:216] Machine: {Timestamp:2026-07-29 08:18:00.039257639 +0000 UTC m=+0.352438491 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098309 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ee29a3a976b782074e1308d679cb2 SystemUUID:ec2ee29a-3a97-6b78-2074-e1308d679cb2 BootID:c4a1636e-5b15-4642-bbd7-b4853d385fba Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:99:00:d4:dd:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:99:00:d4:dd:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:41:fb:ef:73:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 08:18:00.040580 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.040570 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 08:18:00.040748 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.040675 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 08:18:00.041250 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.041231 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 08:18:00.044490 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.044458 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 08:18:00.044652 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.044492 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-90.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 08:18:00.045338 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.045328 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 08:18:00.045369 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.045341 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 08:18:00.045369 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.045349 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 08:18:00.046254 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.046243 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 08:18:00.047629 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.047607 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 08:18:00.051281 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.051265 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 29 08:18:00.051433 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.051423 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 08:18:00.051474 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.051455 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 29 08:18:00.051506 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.051475 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 08:18:00.051506 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.051491 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 08:18:00.051506 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.051503 2582 kubelet.go:398] "Adding apiserver pod source" Jul 29 08:18:00.051694 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.051512 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 08:18:00.051948 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.051935 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 08:18:00.052010 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.051962 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 08:18:00.054528 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.054506 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 08:18:00.056157 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056141 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 08:18:00.056233 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056173 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 08:18:00.056832 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056817 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 08:18:00.056876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056838 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 08:18:00.056876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056851 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 08:18:00.056876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056857 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 08:18:00.056876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056865 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 08:18:00.056876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056871 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 08:18:00.056876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056878 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 08:18:00.057039 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056884 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 08:18:00.057039 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056890 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 08:18:00.057039 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056895 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 08:18:00.057039 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056908 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 08:18:00.057039 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.056914 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 08:18:00.058564 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.058552 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 08:18:00.058564 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.058563 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 08:18:00.062295 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.062258 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-90.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 08:18:00.062295 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.062262 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 08:18:00.062424 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.062263 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 08:18:00.063025 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.063011 2582 server.go:1268] "Started kubelet" Jul 29 08:18:00.063100 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.063073 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 08:18:00.063370 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.063154 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 08:18:00.063484 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.063460 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 08:18:00.064075 ip-10-0-131-90 systemd[1]: Started Kubernetes Kubelet. Jul 29 08:18:00.065305 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.065280 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 08:18:00.065489 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.065477 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 29 08:18:00.068692 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.068673 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 08:18:00.068996 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.067579 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-90.ec2.internal.18c6b4c98e028157 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-90.ec2.internal,UID:ip-10-0-131-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-90.ec2.internal,},FirstTimestamp:2026-07-29 08:18:00.062935383 +0000 UTC m=+0.376116233,LastTimestamp:2026-07-29 08:18:00.062935383 +0000 UTC m=+0.376116233,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-90.ec2.internal,}" Jul 29 08:18:00.071702 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.071685 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 08:18:00.071786 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.071682 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 08:18:00.071786 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.071763 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.072261 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.072242 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 08:18:00.072261 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.072262 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 08:18:00.072392 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.072242 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 08:18:00.072462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.072448 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 29 08:18:00.072462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.072461 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 29 08:18:00.073138 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.073123 2582 factory.go:55] Registering systemd factory Jul 29 08:18:00.073206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.073156 2582 factory.go:223] Registration of the systemd container factory successfully Jul 29 08:18:00.073392 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.073377 2582 factory.go:153] Registering CRI-O factory Jul 29 08:18:00.073450 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.073398 2582 factory.go:223] Registration of the crio container factory successfully Jul 29 08:18:00.073480 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.073451 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 08:18:00.073480 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.073477 2582 factory.go:103] Registering Raw factory Jul 29 08:18:00.073563 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.073493 2582 manager.go:1196] Started watching for new ooms in manager Jul 29 08:18:00.074200 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.074179 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p8z5b" Jul 29 08:18:00.074282 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.074269 2582 manager.go:349] Starting recovery of all containers Jul 29 08:18:00.081856 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.081832 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p8z5b" Jul 29 08:18:00.082568 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.082543 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 08:18:00.092171 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.091938 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 08:18:00.094142 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.094121 2582 manager.go:354] Recovery completed Jul 29 08:18:00.099538 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.099520 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:18:00.103142 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.103123 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:18:00.103213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.103156 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:18:00.103213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.103166 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:18:00.103847 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.103820 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 08:18:00.103847 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.103837 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 08:18:00.103847 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.103856 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 29 08:18:00.103847 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.103824 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:18:00.103847 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.104554 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:18:00.103847 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.104571 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:18:00.103847 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.104580 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:18:00.111759 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.111739 2582 policy_none.go:49] "None policy: Start" Jul 29 08:18:00.111800 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.111765 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 08:18:00.111800 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.111777 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 08:18:00.113105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.113094 2582 policy_none.go:47] "Start" Jul 29 08:18:00.156381 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.156358 2582 manager.go:337] "Starting Device Plugin manager" Jul 29 08:18:00.156498 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.156417 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 08:18:00.156498 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.156430 2582 server.go:87] "Starting device plugin registration server" Jul 29 08:18:00.156751 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.156737 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 08:18:00.156818 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.156752 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 08:18:00.156877 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.156815 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 08:18:00.156921 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.156890 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 08:18:00.156921 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.156898 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 08:18:00.157583 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.157552 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 08:18:00.157705 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.157655 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.243575 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.243488 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 08:18:00.244958 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.244937 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 08:18:00.245038 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.244991 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 08:18:00.245038 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.245027 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 08:18:00.245118 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.245070 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 08:18:00.247131 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.247112 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:18:00.257197 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.257181 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:18:00.258089 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.258073 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:18:00.258174 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.258104 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:18:00.258174 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.258114 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:18:00.258174 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.258139 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.264495 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.264480 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.264562 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.264503 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-90.ec2.internal\": node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.281811 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.281788 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.345968 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.345916 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal"] Jul 29 08:18:00.346115 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.346008 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:18:00.347840 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.347824 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:18:00.347913 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.347856 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:18:00.347913 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.347865 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:18:00.349094 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.349082 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:18:00.349247 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.349232 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.349309 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.349266 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:18:00.349857 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.349840 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:18:00.349928 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.349871 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:18:00.349928 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.349881 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:18:00.350726 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.350709 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:18:00.350799 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.350742 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:18:00.350799 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.350757 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:18:00.352101 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.352087 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.352150 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.352114 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:18:00.352875 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.352861 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:18:00.352963 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.352887 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:18:00.352963 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.352903 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:18:00.373757 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.373726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a35ab5c768d8e59545c7458cdd8d9519-config\") pod \"kube-apiserver-proxy-ip-10-0-131-90.ec2.internal\" (UID: \"a35ab5c768d8e59545c7458cdd8d9519\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.373757 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.373758 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/634c9b31c6b9ff9511da29a8938fb017-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal\" (UID: \"634c9b31c6b9ff9511da29a8938fb017\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.373942 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.373778 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/634c9b31c6b9ff9511da29a8938fb017-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal\" (UID: \"634c9b31c6b9ff9511da29a8938fb017\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.379451 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.379434 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-90.ec2.internal\" not found" node="ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.382014 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.381995 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.384117 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.384100 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-90.ec2.internal\" not found" node="ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.474345 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.474309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a35ab5c768d8e59545c7458cdd8d9519-config\") pod \"kube-apiserver-proxy-ip-10-0-131-90.ec2.internal\" (UID: \"a35ab5c768d8e59545c7458cdd8d9519\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.474345 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.474341 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/634c9b31c6b9ff9511da29a8938fb017-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal\" (UID: \"634c9b31c6b9ff9511da29a8938fb017\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.474545 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.474361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/634c9b31c6b9ff9511da29a8938fb017-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal\" (UID: \"634c9b31c6b9ff9511da29a8938fb017\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.474545 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.474359 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a35ab5c768d8e59545c7458cdd8d9519-config\") pod \"kube-apiserver-proxy-ip-10-0-131-90.ec2.internal\" (UID: \"a35ab5c768d8e59545c7458cdd8d9519\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.474545 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.474399 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/634c9b31c6b9ff9511da29a8938fb017-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal\" (UID: \"634c9b31c6b9ff9511da29a8938fb017\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.474545 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.474425 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/634c9b31c6b9ff9511da29a8938fb017-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal\" (UID: \"634c9b31c6b9ff9511da29a8938fb017\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.482123 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.482102 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.583249 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.583185 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.681748 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.681715 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.683824 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.683805 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.688189 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.688172 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" Jul 29 08:18:00.783957 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.783916 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.884958 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.884884 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.985921 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:00.985881 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:00.997015 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:00.996983 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:18:01.009985 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.009949 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 08:18:01.010151 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.010125 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 08:18:01.010217 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.010178 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 08:18:01.087123 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:01.087082 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:01.096659 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.096598 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 08:13:00 +0000 UTC" deadline="2028-02-15 04:16:36.656932693 +0000 UTC" Jul 29 08:18:01.096659 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.096652 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13579h58m35.560284056s" Jul 29 08:18:01.188313 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:01.188231 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-90.ec2.internal\" not found" Jul 29 08:18:01.224248 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.224219 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:18:01.272640 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.272589 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal" Jul 29 08:18:01.285382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.285352 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 08:18:01.287714 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.287697 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" Jul 29 08:18:01.301965 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.301919 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 08:18:01.303132 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.303115 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:18:01.330581 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:01.330526 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35ab5c768d8e59545c7458cdd8d9519.slice/crio-a95ee314b0cdea01912c8c3e37949fba17f2ea541914fb02813ccadfb64433c0 WatchSource:0}: Error finding container a95ee314b0cdea01912c8c3e37949fba17f2ea541914fb02813ccadfb64433c0: Status 404 returned error can't find the container with id a95ee314b0cdea01912c8c3e37949fba17f2ea541914fb02813ccadfb64433c0 Jul 29 08:18:01.331258 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:01.331228 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634c9b31c6b9ff9511da29a8938fb017.slice/crio-1c336860bf6d5f581c1f4b9b0b677c2d28562b0af2a0319ee5a7b9649c72ba27 WatchSource:0}: Error finding container 1c336860bf6d5f581c1f4b9b0b677c2d28562b0af2a0319ee5a7b9649c72ba27: Status 404 returned error can't find the container with id 1c336860bf6d5f581c1f4b9b0b677c2d28562b0af2a0319ee5a7b9649c72ba27 Jul 29 08:18:01.335904 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:01.335882 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:18:02.055500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.055464 2582 apiserver.go:52] "Watching apiserver" Jul 29 08:18:02.062656 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.062620 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 08:18:02.064104 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.064073 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-c9tjk","kube-system/konnectivity-agent-gd25w","kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal","openshift-cluster-node-tuning-operator/tuned-585dp","openshift-dns/node-resolver-cwbns","openshift-image-registry/node-ca-5vj99","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal","openshift-network-operator/iptables-alerter-8sqqv","openshift-ovn-kubernetes/ovnkube-node-sx2cm","kube-system/global-pull-secret-syncer-mcpcq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck","openshift-multus/multus-additional-cni-plugins-j5cxr","openshift-multus/multus-g2sxx","openshift-multus/network-metrics-daemon-5pfnw"] Jul 29 08:18:02.066162 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.066137 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:02.066311 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.066284 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:02.067809 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.067788 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.069399 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.069375 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:02.069496 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.069456 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:02.070077 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.070000 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-77cpm\"" Jul 29 08:18:02.070190 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.070137 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 08:18:02.070190 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.070151 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 08:18:02.070295 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.070151 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 08:18:02.070295 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.070254 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 08:18:02.071249 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.071224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.072649 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.072628 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 08:18:02.072978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.072955 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.073485 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.073463 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:18:02.073583 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.073489 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-99w5f\"" Jul 29 08:18:02.073583 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.073524 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 08:18:02.074125 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.074106 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.075315 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.075293 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r9xzf\"" Jul 29 08:18:02.075315 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.075305 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 08:18:02.075917 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.075892 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 08:18:02.076166 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.076142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.076330 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.076312 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 08:18:02.076653 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.076634 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 08:18:02.076751 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.076728 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pvrhm\"" Jul 29 08:18:02.076990 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.076970 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 08:18:02.077717 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.077693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.080846 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.080059 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 08:18:02.080846 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.080366 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 08:18:02.080846 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.080428 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 08:18:02.080846 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.080441 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 08:18:02.080846 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.080619 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 08:18:02.080846 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.080647 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 08:18:02.080846 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.080678 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:18:02.081336 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.081005 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tq5dh\"" Jul 29 08:18:02.081336 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.081081 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 08:18:02.081434 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.081353 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s5ghw\"" Jul 29 08:18:02.081434 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.081421 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 08:18:02.082760 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.082727 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:02.084381 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-system-cni-dir\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.084499 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084390 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-cni-dir\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.084499 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084409 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-kubernetes\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.084499 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084436 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-sysctl-d\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.084499 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084459 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79567eb3-7610-4e72-b625-6700f758a78a-tmp-dir\") pod \"node-resolver-cwbns\" (UID: \"79567eb3-7610-4e72-b625-6700f758a78a\") " pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.084740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084507 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:02.084740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-sys\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.084740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084652 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-var-lib-kubelet\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.084740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084689 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-tuned\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.084740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084717 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-run-k8s-cni-cncf-io\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084745 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084779 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b9ccf77-56ae-4384-897d-44864edecb7b-cni-binary-copy\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084811 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-conf-dir\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084787 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084852 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-daemon-config\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084876 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-run\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-lib-modules\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.084908 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084922 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79567eb3-7610-4e72-b625-6700f758a78a-hosts-file\") pod \"node-resolver-cwbns\" (UID: \"79567eb3-7610-4e72-b625-6700f758a78a\") " pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-cnibin\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.084968 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-socket-dir-parent\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085007 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkft6\" (UniqueName: \"kubernetes.io/projected/79567eb3-7610-4e72-b625-6700f758a78a-kube-api-access-kkft6\") pod \"node-resolver-cwbns\" (UID: \"79567eb3-7610-4e72-b625-6700f758a78a\") " pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085057 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-dbus\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085094 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r87n\" (UniqueName: \"kubernetes.io/projected/4b9ccf77-56ae-4384-897d-44864edecb7b-kube-api-access-6r87n\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-modprobe-d\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085149 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvfxr\" (UniqueName: \"kubernetes.io/projected/b96ceeee-07bd-4a3a-9792-149aacc416b6-kube-api-access-zvfxr\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085173 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-kubelet-config\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085213 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-os-release\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085232 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-var-lib-cni-multus\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085248 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-run-multus-certs\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085264 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-sysctl-conf\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085290 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-host\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzn2p\" (UniqueName: \"kubernetes.io/projected/30908914-7c33-4a7b-832a-bbf63c72b09b-kube-api-access-pzn2p\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085337 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-systemd\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96ceeee-07bd-4a3a-9792-149aacc416b6-tmp\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085376 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-run-netns\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-var-lib-cni-bin\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085408 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-var-lib-kubelet\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.085772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-hostroot\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.086751 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-etc-kubernetes\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.086751 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085469 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-sysconfig\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.086751 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.085982 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 08:18:02.086751 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.086200 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 08:18:02.086751 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.086327 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m64ln\"" Jul 29 08:18:02.086751 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.086493 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.088334 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.088313 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.088730 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.088711 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 08:18:02.089107 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.089089 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 08:18:02.090047 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.090025 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 08:18:02.090746 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.090724 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-srps9\"" Jul 29 08:18:02.091101 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.090789 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 08:18:02.091101 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.090796 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 08:18:02.091101 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.090894 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rvt2r\"" Jul 29 08:18:02.100139 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.100117 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 08:18:02.126991 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.126960 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zfwvz" Jul 29 08:18:02.139301 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.139276 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zfwvz" Jul 29 08:18:02.173691 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.173654 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 08:18:02.186130 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186090 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-dbus\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:02.186130 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186146 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r87n\" (UniqueName: \"kubernetes.io/projected/4b9ccf77-56ae-4384-897d-44864edecb7b-kube-api-access-6r87n\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.186420 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-modprobe-d\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.186420 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186249 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzw47\" (UniqueName: \"kubernetes.io/projected/8971e266-0ba5-4a9f-a516-83954d7c971a-kube-api-access-rzw47\") pod \"node-ca-5vj99\" (UID: \"8971e266-0ba5-4a9f-a516-83954d7c971a\") " pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.186420 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzldl\" (UniqueName: \"kubernetes.io/projected/4d066258-79b7-4768-b5f2-b11990e7d37e-kube-api-access-tzldl\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.186420 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-registration-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.186420 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186315 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-dbus\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:02.186420 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186315 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-etc-selinux\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.186789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-var-lib-cni-multus\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.186789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186473 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-host\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.186789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186515 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-cni-bin\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.186789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186543 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-system-cni-dir\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.186789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186570 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-os-release\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.186789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186606 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.186789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186633 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9ldc\" (UniqueName: \"kubernetes.io/projected/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-kube-api-access-c9ldc\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.186789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186712 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzn2p\" (UniqueName: \"kubernetes.io/projected/30908914-7c33-4a7b-832a-bbf63c72b09b-kube-api-access-pzn2p\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:02.186789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186745 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-var-lib-cni-multus\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.186789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186752 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-slash\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186801 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-modprobe-d\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186802 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-var-lib-openvswitch\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186803 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-host\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186835 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbxp9\" (UniqueName: \"kubernetes.io/projected/943bc143-014f-4b30-8fa2-181d33f03bc0-kube-api-access-hbxp9\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186885 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-run-netns\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186907 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-var-lib-cni-bin\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.186956 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-run-netns\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-var-lib-kubelet\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187060 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-etc-kubernetes\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187094 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-var-lib-kubelet\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187099 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63-host-slash\") pod \"iptables-alerter-8sqqv\" (UID: \"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63\") " pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.187180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-etc-kubernetes\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-systemd-units\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187235 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-run-ovn\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187264 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-cni-netd\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-system-cni-dir\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187371 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-system-cni-dir\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187391 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-var-lib-cni-bin\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187401 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-cni-dir\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187429 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-kubernetes\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187448 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-cni-dir\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79567eb3-7610-4e72-b625-6700f758a78a-tmp-dir\") pod \"node-resolver-cwbns\" (UID: \"79567eb3-7610-4e72-b625-6700f758a78a\") " pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187473 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5zcl\" (UniqueName: \"kubernetes.io/projected/9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63-kube-api-access-j5zcl\") pod \"iptables-alerter-8sqqv\" (UID: \"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63\") " pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187491 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-run-systemd\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.187591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187583 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-kubernetes\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187614 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-run-openvswitch\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187630 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d066258-79b7-4768-b5f2-b11990e7d37e-ovnkube-config\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d066258-79b7-4768-b5f2-b11990e7d37e-ovnkube-script-lib\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187703 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-device-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b9ccf77-56ae-4384-897d-44864edecb7b-cni-binary-copy\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-conf-dir\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-daemon-config\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-run\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79567eb3-7610-4e72-b625-6700f758a78a-tmp-dir\") pod \"node-resolver-cwbns\" (UID: \"79567eb3-7610-4e72-b625-6700f758a78a\") " pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187880 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187893 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-run\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d066258-79b7-4768-b5f2-b11990e7d37e-ovn-node-metrics-cert\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187967 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-socket-dir-parent\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.187993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63-iptables-alerter-script\") pod \"iptables-alerter-8sqqv\" (UID: \"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63\") " pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f2d9ce03-0925-43ae-9a2a-f4a6bb186df0-konnectivity-ca\") pod \"konnectivity-agent-gd25w\" (UID: \"f2d9ce03-0925-43ae-9a2a-f4a6bb186df0\") " pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:02.188198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvfxr\" (UniqueName: \"kubernetes.io/projected/b96ceeee-07bd-4a3a-9792-149aacc416b6-kube-api-access-zvfxr\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-etc-openvswitch\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188149 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-kubelet-config\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-os-release\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188223 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-run-multus-certs\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-daemon-config\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188251 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-sysctl-conf\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-systemd\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188290 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-kubelet-config\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96ceeee-07bd-4a3a-9792-149aacc416b6-tmp\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8971e266-0ba5-4a9f-a516-83954d7c971a-host\") pod \"node-ca-5vj99\" (UID: \"8971e266-0ba5-4a9f-a516-83954d7c971a\") " pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188342 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-systemd\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188345 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-sys-fs\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188419 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-socket-dir-parent\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188426 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b9ccf77-56ae-4384-897d-44864edecb7b-cni-binary-copy\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.189059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-hostroot\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-sysconfig\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188487 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-multus-conf-dir\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188495 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-socket-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188512 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-cnibin\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-hostroot\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-sysctl-d\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-run-multus-certs\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188682 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-os-release\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188758 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-sysctl-d\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.188771 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188729 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-sys\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.188816 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-sysconfig\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.188848 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs podName:30908914-7c33-4a7b-832a-bbf63c72b09b nodeName:}" failed. No retries permitted until 2026-07-29 08:18:02.68882158 +0000 UTC m=+3.002002439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs") pod "network-metrics-daemon-5pfnw" (UID: "30908914-7c33-4a7b-832a-bbf63c72b09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.189143 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-sys\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.189206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-var-lib-kubelet\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.189916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.189459 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-sysctl-conf\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.189484 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-tuned\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.189774 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.189794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-var-lib-kubelet\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.189875 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8971e266-0ba5-4a9f-a516-83954d7c971a-serviceca\") pod \"node-ca-5vj99\" (UID: \"8971e266-0ba5-4a9f-a516-83954d7c971a\") " pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.189946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d066258-79b7-4768-b5f2-b11990e7d37e-env-overrides\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.189989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvsx\" (UniqueName: \"kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx\") pod \"network-check-target-c9tjk\" (UID: \"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd\") " pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190063 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkft6\" (UniqueName: \"kubernetes.io/projected/79567eb3-7610-4e72-b625-6700f758a78a-kube-api-access-kkft6\") pod \"node-resolver-cwbns\" (UID: \"79567eb3-7610-4e72-b625-6700f758a78a\") " pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190227 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-kubelet\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-run-netns\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-run-k8s-cni-cncf-io\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-lib-modules\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79567eb3-7610-4e72-b625-6700f758a78a-hosts-file\") pod \"node-resolver-cwbns\" (UID: \"79567eb3-7610-4e72-b625-6700f758a78a\") " pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.190791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190789 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-run-ovn-kubernetes\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.191474 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-cnibin\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.191474 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f2d9ce03-0925-43ae-9a2a-f4a6bb186df0-agent-certs\") pod \"konnectivity-agent-gd25w\" (UID: \"f2d9ce03-0925-43ae-9a2a-f4a6bb186df0\") " pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:02.191474 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.190888 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-node-log\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.191474 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.191072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-log-socket\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.191474 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.191304 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b96ceeee-07bd-4a3a-9792-149aacc416b6-lib-modules\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.191951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.191854 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79567eb3-7610-4e72-b625-6700f758a78a-hosts-file\") pod \"node-resolver-cwbns\" (UID: \"79567eb3-7610-4e72-b625-6700f758a78a\") " pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.191951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.191927 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-host-run-k8s-cni-cncf-io\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.192056 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.192020 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:02.192136 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.192116 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret podName:ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f nodeName:}" failed. No retries permitted until 2026-07-29 08:18:02.692097677 +0000 UTC m=+3.005278515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret") pod "global-pull-secret-syncer-mcpcq" (UID: "ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:02.193658 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.192394 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b9ccf77-56ae-4384-897d-44864edecb7b-cnibin\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.193977 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.193929 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b96ceeee-07bd-4a3a-9792-149aacc416b6-etc-tuned\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.197252 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.197225 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96ceeee-07bd-4a3a-9792-149aacc416b6-tmp\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.198559 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.198535 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r87n\" (UniqueName: \"kubernetes.io/projected/4b9ccf77-56ae-4384-897d-44864edecb7b-kube-api-access-6r87n\") pod \"multus-g2sxx\" (UID: \"4b9ccf77-56ae-4384-897d-44864edecb7b\") " pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.202043 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.202018 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvfxr\" (UniqueName: \"kubernetes.io/projected/b96ceeee-07bd-4a3a-9792-149aacc416b6-kube-api-access-zvfxr\") pod \"tuned-585dp\" (UID: \"b96ceeee-07bd-4a3a-9792-149aacc416b6\") " pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.212148 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.212111 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkft6\" (UniqueName: \"kubernetes.io/projected/79567eb3-7610-4e72-b625-6700f758a78a-kube-api-access-kkft6\") pod \"node-resolver-cwbns\" (UID: \"79567eb3-7610-4e72-b625-6700f758a78a\") " pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.213002 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.212907 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzn2p\" (UniqueName: \"kubernetes.io/projected/30908914-7c33-4a7b-832a-bbf63c72b09b-kube-api-access-pzn2p\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:02.254468 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.254403 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" event={"ID":"634c9b31c6b9ff9511da29a8938fb017","Type":"ContainerStarted","Data":"1c336860bf6d5f581c1f4b9b0b677c2d28562b0af2a0319ee5a7b9649c72ba27"} Jul 29 08:18:02.255934 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.255850 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal" event={"ID":"a35ab5c768d8e59545c7458cdd8d9519","Type":"ContainerStarted","Data":"a95ee314b0cdea01912c8c3e37949fba17f2ea541914fb02813ccadfb64433c0"} Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.291848 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzw47\" (UniqueName: \"kubernetes.io/projected/8971e266-0ba5-4a9f-a516-83954d7c971a-kube-api-access-rzw47\") pod \"node-ca-5vj99\" (UID: \"8971e266-0ba5-4a9f-a516-83954d7c971a\") " pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.291897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzldl\" (UniqueName: \"kubernetes.io/projected/4d066258-79b7-4768-b5f2-b11990e7d37e-kube-api-access-tzldl\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.291919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-registration-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.291937 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-etc-selinux\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.291956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-cni-bin\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.291973 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-system-cni-dir\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.291998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-os-release\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292018 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9ldc\" (UniqueName: \"kubernetes.io/projected/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-kube-api-access-c9ldc\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-slash\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292071 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-var-lib-openvswitch\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292093 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbxp9\" (UniqueName: \"kubernetes.io/projected/943bc143-014f-4b30-8fa2-181d33f03bc0-kube-api-access-hbxp9\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63-host-slash\") pod \"iptables-alerter-8sqqv\" (UID: \"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63\") " pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-systemd-units\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-run-ovn\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-cni-netd\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.294134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5zcl\" (UniqueName: \"kubernetes.io/projected/9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63-kube-api-access-j5zcl\") pod \"iptables-alerter-8sqqv\" (UID: \"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63\") " pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292212 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-run-systemd\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292264 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-run-systemd\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292500 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-slash\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292543 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-var-lib-openvswitch\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292773 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63-host-slash\") pod \"iptables-alerter-8sqqv\" (UID: \"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63\") " pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292819 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-systemd-units\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-run-ovn\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292879 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-cni-netd\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-run-openvswitch\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.292954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d066258-79b7-4768-b5f2-b11990e7d37e-ovnkube-config\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293020 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d066258-79b7-4768-b5f2-b11990e7d37e-ovnkube-script-lib\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-device-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d066258-79b7-4768-b5f2-b11990e7d37e-ovn-node-metrics-cert\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293140 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.296406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63-iptables-alerter-script\") pod \"iptables-alerter-8sqqv\" (UID: \"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63\") " pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f2d9ce03-0925-43ae-9a2a-f4a6bb186df0-konnectivity-ca\") pod \"konnectivity-agent-gd25w\" (UID: \"f2d9ce03-0925-43ae-9a2a-f4a6bb186df0\") " pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-etc-openvswitch\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293308 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-registration-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293317 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8971e266-0ba5-4a9f-a516-83954d7c971a-host\") pod \"node-ca-5vj99\" (UID: \"8971e266-0ba5-4a9f-a516-83954d7c971a\") " pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-sys-fs\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293370 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8971e266-0ba5-4a9f-a516-83954d7c971a-host\") pod \"node-ca-5vj99\" (UID: \"8971e266-0ba5-4a9f-a516-83954d7c971a\") " pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293429 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-socket-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293451 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-os-release\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.293458 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-cnibin\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294017 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d066258-79b7-4768-b5f2-b11990e7d37e-ovnkube-config\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294088 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8971e266-0ba5-4a9f-a516-83954d7c971a-serviceca\") pod \"node-ca-5vj99\" (UID: \"8971e266-0ba5-4a9f-a516-83954d7c971a\") " pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294118 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d066258-79b7-4768-b5f2-b11990e7d37e-env-overrides\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294160 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvsx\" (UniqueName: \"kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx\") pod \"network-check-target-c9tjk\" (UID: \"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd\") " pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.297378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-kubelet\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-run-netns\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294252 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-run-ovn-kubernetes\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294287 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f2d9ce03-0925-43ae-9a2a-f4a6bb186df0-agent-certs\") pod \"konnectivity-agent-gd25w\" (UID: \"f2d9ce03-0925-43ae-9a2a-f4a6bb186df0\") " pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-node-log\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294341 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-log-socket\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294455 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-log-socket\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.294971 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-run-netns\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295026 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d066258-79b7-4768-b5f2-b11990e7d37e-ovnkube-script-lib\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295041 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-cni-bin\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295092 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-node-log\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295106 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-etc-selinux\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-cnibin\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295143 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-socket-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295165 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-sys-fs\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295173 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-run-openvswitch\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295186 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.298206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295225 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-run-ovn-kubernetes\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-device-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295311 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-system-cni-dir\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295448 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8971e266-0ba5-4a9f-a516-83954d7c971a-serviceca\") pod \"node-ca-5vj99\" (UID: \"8971e266-0ba5-4a9f-a516-83954d7c971a\") " pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295674 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d066258-79b7-4768-b5f2-b11990e7d37e-env-overrides\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295715 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295774 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-host-kubelet\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295820 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d066258-79b7-4768-b5f2-b11990e7d37e-etc-openvswitch\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295874 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/943bc143-014f-4b30-8fa2-181d33f03bc0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295909 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f2d9ce03-0925-43ae-9a2a-f4a6bb186df0-konnectivity-ca\") pod \"konnectivity-agent-gd25w\" (UID: \"f2d9ce03-0925-43ae-9a2a-f4a6bb186df0\") " pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.295917 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.296197 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.299165 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.296346 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63-iptables-alerter-script\") pod \"iptables-alerter-8sqqv\" (UID: \"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63\") " pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.302918 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.302847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d066258-79b7-4768-b5f2-b11990e7d37e-ovn-node-metrics-cert\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.307379 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.307054 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:18:02.307379 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.307081 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:18:02.307379 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.307098 2582 projected.go:196] Error preparing data for projected volume kube-api-access-jqvsx for pod openshift-network-diagnostics/network-check-target-c9tjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:02.307379 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.307179 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx podName:c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd nodeName:}" failed. No retries permitted until 2026-07-29 08:18:02.807155587 +0000 UTC m=+3.120336451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvsx" (UniqueName: "kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx") pod "network-check-target-c9tjk" (UID: "c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:02.309374 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.309346 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f2d9ce03-0925-43ae-9a2a-f4a6bb186df0-agent-certs\") pod \"konnectivity-agent-gd25w\" (UID: \"f2d9ce03-0925-43ae-9a2a-f4a6bb186df0\") " pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:02.310230 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.310206 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzw47\" (UniqueName: \"kubernetes.io/projected/8971e266-0ba5-4a9f-a516-83954d7c971a-kube-api-access-rzw47\") pod \"node-ca-5vj99\" (UID: \"8971e266-0ba5-4a9f-a516-83954d7c971a\") " pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.310332 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.310248 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbxp9\" (UniqueName: \"kubernetes.io/projected/943bc143-014f-4b30-8fa2-181d33f03bc0-kube-api-access-hbxp9\") pod \"aws-ebs-csi-driver-node-qz9ck\" (UID: \"943bc143-014f-4b30-8fa2-181d33f03bc0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.311131 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.311088 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9ldc\" (UniqueName: \"kubernetes.io/projected/d5c50b1e-53b1-4dab-87ed-2b246ce43f74-kube-api-access-c9ldc\") pod \"multus-additional-cni-plugins-j5cxr\" (UID: \"d5c50b1e-53b1-4dab-87ed-2b246ce43f74\") " pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.312563 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.312538 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5zcl\" (UniqueName: \"kubernetes.io/projected/9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63-kube-api-access-j5zcl\") pod \"iptables-alerter-8sqqv\" (UID: \"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63\") " pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.312731 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.312689 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzldl\" (UniqueName: \"kubernetes.io/projected/4d066258-79b7-4768-b5f2-b11990e7d37e-kube-api-access-tzldl\") pod \"ovnkube-node-sx2cm\" (UID: \"4d066258-79b7-4768-b5f2-b11990e7d37e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.342142 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.342103 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:18:02.380730 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.380694 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g2sxx" Jul 29 08:18:02.390476 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.390443 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-585dp" Jul 29 08:18:02.399122 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.398856 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cwbns" Jul 29 08:18:02.405458 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.405433 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5vj99" Jul 29 08:18:02.413803 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.413780 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:02.423305 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.423278 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8sqqv" Jul 29 08:18:02.430445 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.430423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:02.438427 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.438403 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" Jul 29 08:18:02.445346 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.445295 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" Jul 29 08:18:02.698528 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.698437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:02.698528 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.698487 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:02.698806 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.698624 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:02.698806 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.698625 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:02.698806 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.698712 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs podName:30908914-7c33-4a7b-832a-bbf63c72b09b nodeName:}" failed. No retries permitted until 2026-07-29 08:18:03.698696845 +0000 UTC m=+4.011877691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs") pod "network-metrics-daemon-5pfnw" (UID: "30908914-7c33-4a7b-832a-bbf63c72b09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:02.698806 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.698728 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret podName:ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f nodeName:}" failed. No retries permitted until 2026-07-29 08:18:03.69872095 +0000 UTC m=+4.011901789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret") pod "global-pull-secret-syncer-mcpcq" (UID: "ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:02.899885 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:02.899852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvsx\" (UniqueName: \"kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx\") pod \"network-check-target-c9tjk\" (UID: \"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd\") " pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:02.900077 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.900055 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:18:02.900077 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.900077 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:18:02.900164 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.900088 2582 projected.go:196] Error preparing data for projected volume kube-api-access-jqvsx for pod openshift-network-diagnostics/network-check-target-c9tjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:02.900164 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:02.900159 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx podName:c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd nodeName:}" failed. No retries permitted until 2026-07-29 08:18:03.900141116 +0000 UTC m=+4.213321977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvsx" (UniqueName: "kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx") pod "network-check-target-c9tjk" (UID: "c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:03.142083 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:03.142034 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 08:13:02 +0000 UTC" deadline="2028-03-12 13:08:02.867350506 +0000 UTC" Jul 29 08:18:03.142083 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:03.142073 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14212h49m59.725280366s" Jul 29 08:18:03.246269 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:03.246229 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:03.246440 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:03.246362 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:03.361086 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:03.361048 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96ceeee_07bd_4a3a_9792_149aacc416b6.slice/crio-640f110248366f133bd0095a673b91a5e7f04d795cb019f731cb64af8357002a WatchSource:0}: Error finding container 640f110248366f133bd0095a673b91a5e7f04d795cb019f731cb64af8357002a: Status 404 returned error can't find the container with id 640f110248366f133bd0095a673b91a5e7f04d795cb019f731cb64af8357002a Jul 29 08:18:03.380177 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:03.380141 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79567eb3_7610_4e72_b625_6700f758a78a.slice/crio-3b5484ee9c9de63abcfa6b681d627773cbe42341208c61cf74eae95408d617be WatchSource:0}: Error finding container 3b5484ee9c9de63abcfa6b681d627773cbe42341208c61cf74eae95408d617be: Status 404 returned error can't find the container with id 3b5484ee9c9de63abcfa6b681d627773cbe42341208c61cf74eae95408d617be Jul 29 08:18:03.380586 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:03.380561 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943bc143_014f_4b30_8fa2_181d33f03bc0.slice/crio-b7b33b4b993e942bee27216b1390c7a144bd3d1a09d01699f80f7382ab705ead WatchSource:0}: Error finding container b7b33b4b993e942bee27216b1390c7a144bd3d1a09d01699f80f7382ab705ead: Status 404 returned error can't find the container with id b7b33b4b993e942bee27216b1390c7a144bd3d1a09d01699f80f7382ab705ead Jul 29 08:18:03.381148 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:03.381068 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d9ce03_0925_43ae_9a2a_f4a6bb186df0.slice/crio-4196cbb9a5bc7a9b459998fcdc859148484858fac30fc87049eaf00c7e72e2bf WatchSource:0}: Error finding container 4196cbb9a5bc7a9b459998fcdc859148484858fac30fc87049eaf00c7e72e2bf: Status 404 returned error can't find the container with id 4196cbb9a5bc7a9b459998fcdc859148484858fac30fc87049eaf00c7e72e2bf Jul 29 08:18:03.381968 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:03.381940 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9ccf77_56ae_4384_897d_44864edecb7b.slice/crio-2ce43ec89c76a0d55b8d243dcf4c4a4d5c5060682d33cee7ed14ad7fd8ac97e7 WatchSource:0}: Error finding container 2ce43ec89c76a0d55b8d243dcf4c4a4d5c5060682d33cee7ed14ad7fd8ac97e7: Status 404 returned error can't find the container with id 2ce43ec89c76a0d55b8d243dcf4c4a4d5c5060682d33cee7ed14ad7fd8ac97e7 Jul 29 08:18:03.383254 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:03.383232 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d066258_79b7_4768_b5f2_b11990e7d37e.slice/crio-2982f28f59c067ef2e0e109a008d293b618cbf0af03532ec2ce3a5960d98f1f5 WatchSource:0}: Error finding container 2982f28f59c067ef2e0e109a008d293b618cbf0af03532ec2ce3a5960d98f1f5: Status 404 returned error can't find the container with id 2982f28f59c067ef2e0e109a008d293b618cbf0af03532ec2ce3a5960d98f1f5 Jul 29 08:18:03.384928 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:03.384777 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8971e266_0ba5_4a9f_a516_83954d7c971a.slice/crio-429356899ae69523e4b421e9a0a054bdd82bbcd2434420c58b0b049b38be535f WatchSource:0}: Error finding container 429356899ae69523e4b421e9a0a054bdd82bbcd2434420c58b0b049b38be535f: Status 404 returned error can't find the container with id 429356899ae69523e4b421e9a0a054bdd82bbcd2434420c58b0b049b38be535f Jul 29 08:18:03.386008 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:03.385926 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd6cf4a_6a4b_47d6_b111_f466cc1d1c63.slice/crio-9272e4b30fa96c910d3b11997e5e7cb9ec194337693337ae7dd2da1f88ae76bd WatchSource:0}: Error finding container 9272e4b30fa96c910d3b11997e5e7cb9ec194337693337ae7dd2da1f88ae76bd: Status 404 returned error can't find the container with id 9272e4b30fa96c910d3b11997e5e7cb9ec194337693337ae7dd2da1f88ae76bd Jul 29 08:18:03.387069 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:03.387045 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c50b1e_53b1_4dab_87ed_2b246ce43f74.slice/crio-14fdf34e24c61ca9d910d543b183f683953dcff2d36062f9c0f0f9f029ea0f0e WatchSource:0}: Error finding container 14fdf34e24c61ca9d910d543b183f683953dcff2d36062f9c0f0f9f029ea0f0e: Status 404 returned error can't find the container with id 14fdf34e24c61ca9d910d543b183f683953dcff2d36062f9c0f0f9f029ea0f0e Jul 29 08:18:03.705955 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:03.705819 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:03.705955 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:03.705884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:03.706155 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:03.705992 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:03.706155 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:03.705992 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:03.706155 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:03.706063 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret podName:ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f nodeName:}" failed. No retries permitted until 2026-07-29 08:18:05.706044613 +0000 UTC m=+6.019225454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret") pod "global-pull-secret-syncer-mcpcq" (UID: "ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:03.706155 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:03.706082 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs podName:30908914-7c33-4a7b-832a-bbf63c72b09b nodeName:}" failed. No retries permitted until 2026-07-29 08:18:05.706072561 +0000 UTC m=+6.019253400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs") pod "network-metrics-daemon-5pfnw" (UID: "30908914-7c33-4a7b-832a-bbf63c72b09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:03.829268 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:03.829048 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:18:03.908122 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:03.908083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvsx\" (UniqueName: \"kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx\") pod \"network-check-target-c9tjk\" (UID: \"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd\") " pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:03.908336 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:03.908275 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:18:03.908336 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:03.908298 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:18:03.908336 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:03.908310 2582 projected.go:196] Error preparing data for projected volume kube-api-access-jqvsx for pod openshift-network-diagnostics/network-check-target-c9tjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:03.908487 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:03.908373 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx podName:c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd nodeName:}" failed. No retries permitted until 2026-07-29 08:18:05.90835391 +0000 UTC m=+6.221534765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvsx" (UniqueName: "kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx") pod "network-check-target-c9tjk" (UID: "c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:04.143211 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.143088 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 08:13:02 +0000 UTC" deadline="2028-03-23 01:34:54.265969407 +0000 UTC" Jul 29 08:18:04.143211 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.143162 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14465h16m50.122812396s" Jul 29 08:18:04.246893 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.246856 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:04.247124 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:04.247005 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:04.247124 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.247092 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:04.247259 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:04.247200 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:04.269515 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.269426 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" event={"ID":"d5c50b1e-53b1-4dab-87ed-2b246ce43f74","Type":"ContainerStarted","Data":"14fdf34e24c61ca9d910d543b183f683953dcff2d36062f9c0f0f9f029ea0f0e"} Jul 29 08:18:04.290671 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.290433 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8sqqv" event={"ID":"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63","Type":"ContainerStarted","Data":"9272e4b30fa96c910d3b11997e5e7cb9ec194337693337ae7dd2da1f88ae76bd"} Jul 29 08:18:04.306047 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.306004 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5vj99" event={"ID":"8971e266-0ba5-4a9f-a516-83954d7c971a","Type":"ContainerStarted","Data":"429356899ae69523e4b421e9a0a054bdd82bbcd2434420c58b0b049b38be535f"} Jul 29 08:18:04.316636 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.316576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" event={"ID":"4d066258-79b7-4768-b5f2-b11990e7d37e","Type":"ContainerStarted","Data":"2982f28f59c067ef2e0e109a008d293b618cbf0af03532ec2ce3a5960d98f1f5"} Jul 29 08:18:04.327430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.327392 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gd25w" event={"ID":"f2d9ce03-0925-43ae-9a2a-f4a6bb186df0","Type":"ContainerStarted","Data":"4196cbb9a5bc7a9b459998fcdc859148484858fac30fc87049eaf00c7e72e2bf"} Jul 29 08:18:04.345068 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.345023 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-585dp" event={"ID":"b96ceeee-07bd-4a3a-9792-149aacc416b6","Type":"ContainerStarted","Data":"640f110248366f133bd0095a673b91a5e7f04d795cb019f731cb64af8357002a"} Jul 29 08:18:04.354660 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.354470 2582 generic.go:356] "Generic (PLEG): container finished" podID="634c9b31c6b9ff9511da29a8938fb017" containerID="20a482e6a272d7ad03f7f975bbee5746fe8d204b44cfafc64917f5849cdbf457" exitCode=0 Jul 29 08:18:04.354660 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.354581 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" event={"ID":"634c9b31c6b9ff9511da29a8938fb017","Type":"ContainerDied","Data":"20a482e6a272d7ad03f7f975bbee5746fe8d204b44cfafc64917f5849cdbf457"} Jul 29 08:18:04.364462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.364427 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal" event={"ID":"a35ab5c768d8e59545c7458cdd8d9519","Type":"ContainerStarted","Data":"91a4ed6074830936596bab8299f911e13b327bee48247f7c390593dc641eb45d"} Jul 29 08:18:04.378511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.378394 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g2sxx" event={"ID":"4b9ccf77-56ae-4384-897d-44864edecb7b","Type":"ContainerStarted","Data":"2ce43ec89c76a0d55b8d243dcf4c4a4d5c5060682d33cee7ed14ad7fd8ac97e7"} Jul 29 08:18:04.395140 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.395001 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cwbns" event={"ID":"79567eb3-7610-4e72-b625-6700f758a78a","Type":"ContainerStarted","Data":"3b5484ee9c9de63abcfa6b681d627773cbe42341208c61cf74eae95408d617be"} Jul 29 08:18:04.408874 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:04.408782 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" event={"ID":"943bc143-014f-4b30-8fa2-181d33f03bc0","Type":"ContainerStarted","Data":"b7b33b4b993e942bee27216b1390c7a144bd3d1a09d01699f80f7382ab705ead"} Jul 29 08:18:05.245900 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:05.245862 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:05.246416 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:05.246048 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:05.441828 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:05.441769 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" event={"ID":"634c9b31c6b9ff9511da29a8938fb017","Type":"ContainerStarted","Data":"2f01fe5114f88a15ff1ef119685c9c478b4509428aea6cbf59b902ea0571b5d2"} Jul 29 08:18:05.460347 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:05.460279 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-90.ec2.internal" podStartSLOduration=3.460253949 podStartE2EDuration="3.460253949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:04.394846644 +0000 UTC m=+4.708027508" watchObservedRunningTime="2026-07-29 08:18:05.460253949 +0000 UTC m=+5.773434814" Jul 29 08:18:05.727089 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:05.727049 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:05.727294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:05.727124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:05.727382 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:05.727297 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:05.727382 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:05.727366 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret podName:ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f nodeName:}" failed. No retries permitted until 2026-07-29 08:18:09.727346359 +0000 UTC m=+10.040527202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret") pod "global-pull-secret-syncer-mcpcq" (UID: "ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:05.727845 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:05.727810 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:05.727961 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:05.727913 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs podName:30908914-7c33-4a7b-832a-bbf63c72b09b nodeName:}" failed. No retries permitted until 2026-07-29 08:18:09.727856271 +0000 UTC m=+10.041037133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs") pod "network-metrics-daemon-5pfnw" (UID: "30908914-7c33-4a7b-832a-bbf63c72b09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:05.930106 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:05.929358 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvsx\" (UniqueName: \"kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx\") pod \"network-check-target-c9tjk\" (UID: \"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd\") " pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:05.930106 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:05.929582 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:18:05.930106 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:05.929615 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:18:05.930106 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:05.929630 2582 projected.go:196] Error preparing data for projected volume kube-api-access-jqvsx for pod openshift-network-diagnostics/network-check-target-c9tjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:05.930106 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:05.929715 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx podName:c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd nodeName:}" failed. No retries permitted until 2026-07-29 08:18:09.929693958 +0000 UTC m=+10.242874822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvsx" (UniqueName: "kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx") pod "network-check-target-c9tjk" (UID: "c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:06.246865 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:06.246829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:06.247351 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:06.246985 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:06.247351 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:06.247087 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:06.247351 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:06.247178 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:07.246307 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:07.246272 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:07.246477 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:07.246435 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:08.248736 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:08.248635 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:08.249249 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:08.248742 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:08.249249 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:08.248901 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:08.249249 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:08.249012 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:09.245481 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:09.245448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:09.245680 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:09.245604 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:09.768813 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:09.768065 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:09.768813 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:09.768120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:09.768813 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:09.768263 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:09.768813 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:09.768321 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret podName:ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f nodeName:}" failed. No retries permitted until 2026-07-29 08:18:17.768306428 +0000 UTC m=+18.081487267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret") pod "global-pull-secret-syncer-mcpcq" (UID: "ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:09.768813 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:09.768717 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:09.768813 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:09.768774 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs podName:30908914-7c33-4a7b-832a-bbf63c72b09b nodeName:}" failed. No retries permitted until 2026-07-29 08:18:17.768755442 +0000 UTC m=+18.081936288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs") pod "network-metrics-daemon-5pfnw" (UID: "30908914-7c33-4a7b-832a-bbf63c72b09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:09.970037 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:09.969985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvsx\" (UniqueName: \"kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx\") pod \"network-check-target-c9tjk\" (UID: \"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd\") " pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:09.970260 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:09.970240 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:18:09.970346 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:09.970268 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:18:09.970346 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:09.970281 2582 projected.go:196] Error preparing data for projected volume kube-api-access-jqvsx for pod openshift-network-diagnostics/network-check-target-c9tjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:09.970468 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:09.970353 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx podName:c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd nodeName:}" failed. No retries permitted until 2026-07-29 08:18:17.97033265 +0000 UTC m=+18.283513508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvsx" (UniqueName: "kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx") pod "network-check-target-c9tjk" (UID: "c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:10.247288 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:10.246696 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:10.247288 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:10.246824 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:10.247288 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:10.247147 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:10.247288 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:10.247245 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:11.245650 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:11.245601 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:11.246184 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:11.245783 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:12.245689 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:12.245649 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:12.245689 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:12.245678 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:12.246120 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:12.245805 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:12.246120 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:12.245966 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:13.245530 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:13.245494 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:13.245738 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:13.245677 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:14.245881 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:14.245422 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:14.245881 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:14.245546 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:14.245881 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:14.245644 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:14.245881 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:14.245736 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:15.245662 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:15.245553 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:15.245838 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:15.245694 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:16.245266 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:16.245227 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:16.245725 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:16.245227 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:16.245725 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:16.245345 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:16.245725 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:16.245442 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:17.245785 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:17.245751 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:17.246248 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:17.245892 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:17.832117 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:17.832078 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:17.832379 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:17.832145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:17.832379 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:17.832242 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:17.832379 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:17.832249 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:17.832379 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:17.832310 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs podName:30908914-7c33-4a7b-832a-bbf63c72b09b nodeName:}" failed. No retries permitted until 2026-07-29 08:18:33.832293605 +0000 UTC m=+34.145474443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs") pod "network-metrics-daemon-5pfnw" (UID: "30908914-7c33-4a7b-832a-bbf63c72b09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:17.832379 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:17.832325 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret podName:ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f nodeName:}" failed. No retries permitted until 2026-07-29 08:18:33.832318431 +0000 UTC m=+34.145499269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret") pod "global-pull-secret-syncer-mcpcq" (UID: "ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:18.033745 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:18.033708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvsx\" (UniqueName: \"kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx\") pod \"network-check-target-c9tjk\" (UID: \"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd\") " pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:18.033969 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:18.033870 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:18:18.033969 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:18.033889 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:18:18.033969 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:18.033900 2582 projected.go:196] Error preparing data for projected volume kube-api-access-jqvsx for pod openshift-network-diagnostics/network-check-target-c9tjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:18.033969 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:18.033967 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx podName:c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd nodeName:}" failed. No retries permitted until 2026-07-29 08:18:34.033949146 +0000 UTC m=+34.347129999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvsx" (UniqueName: "kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx") pod "network-check-target-c9tjk" (UID: "c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:18.246076 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:18.245613 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:18.246076 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:18.245674 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:18.246076 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:18.245753 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:18.246076 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:18.245896 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:19.245511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:19.245477 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:19.245721 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:19.245599 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:20.246917 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:20.246881 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:20.247352 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:20.246986 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:20.247352 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:20.247046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:20.247352 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:20.247156 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:21.246184 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:21.246145 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:21.246370 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:21.246287 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:22.245707 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:22.245668 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:22.245707 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:22.245694 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:22.246318 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:22.245818 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:22.246318 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:22.246167 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:23.246033 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.245612 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:23.246033 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:23.245970 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:23.482996 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.482970 2582 generic.go:356] "Generic (PLEG): container finished" podID="d5c50b1e-53b1-4dab-87ed-2b246ce43f74" containerID="a3c49511298b19760d37a5ebc806df888764e864c47cbc70d472775dcae24e65" exitCode=0 Jul 29 08:18:23.483105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.483051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" event={"ID":"d5c50b1e-53b1-4dab-87ed-2b246ce43f74","Type":"ContainerDied","Data":"a3c49511298b19760d37a5ebc806df888764e864c47cbc70d472775dcae24e65"} Jul 29 08:18:23.484574 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.484549 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5vj99" event={"ID":"8971e266-0ba5-4a9f-a516-83954d7c971a","Type":"ContainerStarted","Data":"d019c89d0ff5c2763503ebafd1a90e77ba74a95281abfe07f53eaa2ecdbdf08f"} Jul 29 08:18:23.492470 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.492444 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" event={"ID":"4d066258-79b7-4768-b5f2-b11990e7d37e","Type":"ContainerStarted","Data":"9eaed6ea005151645a1b28b90d1af111c6cea1907e312c2fca7cd692b387f803"} Jul 29 08:18:23.492570 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.492532 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" event={"ID":"4d066258-79b7-4768-b5f2-b11990e7d37e","Type":"ContainerStarted","Data":"0b808f6eb33f3af2c59b70c4bcc26465238af488266b34c6090bd9c5ec626f9b"} Jul 29 08:18:23.492659 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.492583 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" event={"ID":"4d066258-79b7-4768-b5f2-b11990e7d37e","Type":"ContainerStarted","Data":"844adbaff206c7230daac2f3eb7ae2ea20a4aac7a00d273a9b050fee2ba18e0f"} Jul 29 08:18:23.492659 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.492652 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" event={"ID":"4d066258-79b7-4768-b5f2-b11990e7d37e","Type":"ContainerStarted","Data":"08011bebb3d7865e07d8b9aa5187a87b9355d3cff5876e56c09bba5136e178cb"} Jul 29 08:18:23.492762 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.492691 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" event={"ID":"4d066258-79b7-4768-b5f2-b11990e7d37e","Type":"ContainerStarted","Data":"858c56c694d1a93128ee814cb4beb84c8cfad5e805423414dde4581653bfb360"} Jul 29 08:18:23.494285 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.494260 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gd25w" event={"ID":"f2d9ce03-0925-43ae-9a2a-f4a6bb186df0","Type":"ContainerStarted","Data":"b40104d1a56f2002f04fb1510a03c7b15d8fcbb0374a2ae562a62d88a262a6ea"} Jul 29 08:18:23.495708 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.495679 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-585dp" event={"ID":"b96ceeee-07bd-4a3a-9792-149aacc416b6","Type":"ContainerStarted","Data":"308cf1a49bf71378c725fbebed241492638d98346150646d8c35ea297782f9c9"} Jul 29 08:18:23.497071 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.497048 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g2sxx" event={"ID":"4b9ccf77-56ae-4384-897d-44864edecb7b","Type":"ContainerStarted","Data":"56754dd19f9c857f6c70ed1e45b1e3af26b6f3d9e2058f9b4fda19051d2b1106"} Jul 29 08:18:23.498399 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.498380 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cwbns" event={"ID":"79567eb3-7610-4e72-b625-6700f758a78a","Type":"ContainerStarted","Data":"3639dc212e871fa484d519f3d75b9cbcb09484bcbcad628b7869e03999495b57"} Jul 29 08:18:23.499848 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.499830 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" event={"ID":"943bc143-014f-4b30-8fa2-181d33f03bc0","Type":"ContainerStarted","Data":"51a0776478f2ceb760339fc86fcb4b487d4728de07342e60c264db9910c3dc34"} Jul 29 08:18:23.516613 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.516573 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-90.ec2.internal" podStartSLOduration=21.516561063 podStartE2EDuration="21.516561063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:05.461647032 +0000 UTC m=+5.774827899" watchObservedRunningTime="2026-07-29 08:18:23.516561063 +0000 UTC m=+23.829741922" Jul 29 08:18:23.533174 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.533133 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gd25w" podStartSLOduration=7.216503692 podStartE2EDuration="22.533117428s" totalImagesPullingTime="15.316613736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:23.53281456 +0000 UTC m=+23.845995419" watchObservedRunningTime="2026-07-29 08:18:23.533117428 +0000 UTC m=+23.846298288" Jul 29 08:18:23.554660 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.554594 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g2sxx" podStartSLOduration=2.917043177 podStartE2EDuration="22.554580833s" totalImagesPullingTime="19.637537656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:23.553641984 +0000 UTC m=+23.866822845" watchObservedRunningTime="2026-07-29 08:18:23.554580833 +0000 UTC m=+23.867761709" Jul 29 08:18:23.593866 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.593809 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-585dp" podStartSLOduration=2.970860765 podStartE2EDuration="22.593785236s" totalImagesPullingTime="19.622924471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:23.574190853 +0000 UTC m=+23.887371714" watchObservedRunningTime="2026-07-29 08:18:23.593785236 +0000 UTC m=+23.906966097" Jul 29 08:18:23.594013 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.593978 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cwbns" podStartSLOduration=3.012570027 podStartE2EDuration="22.593966636s" totalImagesPullingTime="19.581396609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:23.593943145 +0000 UTC m=+23.907123986" watchObservedRunningTime="2026-07-29 08:18:23.593966636 +0000 UTC m=+23.907147499" Jul 29 08:18:23.611832 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.611789 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5vj99" podStartSLOduration=3.039137925 podStartE2EDuration="22.611770375s" totalImagesPullingTime="19.57263245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:23.611733813 +0000 UTC m=+23.924914674" watchObservedRunningTime="2026-07-29 08:18:23.611770375 +0000 UTC m=+23.924951236" Jul 29 08:18:23.706518 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.706480 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:23.776493 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:23.776406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:24.245528 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:24.245496 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:24.245843 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:24.245546 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:24.245843 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:24.245648 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:24.245843 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:24.245753 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:24.438351 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:24.438325 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 08:18:24.504135 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:24.504094 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" event={"ID":"943bc143-014f-4b30-8fa2-181d33f03bc0","Type":"ContainerStarted","Data":"63f16a598d10db2c8b83833275ea814ddad1ecb1d236d813265b7819f79c0942"} Jul 29 08:18:24.505967 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:24.505930 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8sqqv" event={"ID":"9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63","Type":"ContainerStarted","Data":"0feb95b5b62b5418cf2b72d38b6491ced0a0346bd801e9c166a1b9b06e96e8e5"} Jul 29 08:18:24.509397 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:24.509368 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" event={"ID":"4d066258-79b7-4768-b5f2-b11990e7d37e","Type":"ContainerStarted","Data":"45a631199b86b0ed9c8aa56110c004c3aec8862c8efa8581533de40cefaf48fc"} Jul 29 08:18:24.654402 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:24.654365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:24.724333 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:24.724298 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:25.171605 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:25.171451 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T08:18:24.438344522Z","UUID":"a307a60d-ae90-4079-86d3-9afcaa23ec75","Handler":null,"Name":"","Endpoint":""} Jul 29 08:18:25.173410 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:25.173368 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 08:18:25.173410 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:25.173402 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 08:18:25.246070 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:25.246035 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:25.246231 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:25.246171 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:25.371431 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:25.371391 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:25.372069 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:25.372051 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:25.387514 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:25.387461 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8sqqv" podStartSLOduration=4.804049801 podStartE2EDuration="24.38744116s" totalImagesPullingTime="19.583391359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:24.525176554 +0000 UTC m=+24.838357416" watchObservedRunningTime="2026-07-29 08:18:25.38744116 +0000 UTC m=+25.700622021" Jul 29 08:18:25.512707 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:25.512243 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:25.512707 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:25.512483 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gd25w" Jul 29 08:18:25.582146 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:25.582113 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:26.245276 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:26.245246 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:26.245477 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:26.245374 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:26.245477 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:26.245403 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:26.245596 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:26.245524 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:26.516448 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:26.516357 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" event={"ID":"943bc143-014f-4b30-8fa2-181d33f03bc0","Type":"ContainerStarted","Data":"849de5b4f4936327230e5b068a5fb5107841aae31474a265f5b9d60f6404e71a"} Jul 29 08:18:26.520337 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:26.520306 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" event={"ID":"4d066258-79b7-4768-b5f2-b11990e7d37e","Type":"ContainerStarted","Data":"a1278aa0492a96b84f50a8c4951490adc843dd49173a54b7b01d8e2a0eb44bcc"} Jul 29 08:18:26.536072 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:26.536021 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qz9ck" podStartSLOduration=3.812970346 podStartE2EDuration="25.536006246s" totalImagesPullingTime="21.7230359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:26.535599396 +0000 UTC m=+26.848780254" watchObservedRunningTime="2026-07-29 08:18:26.536006246 +0000 UTC m=+26.849187105" Jul 29 08:18:27.245804 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:27.245593 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:27.245980 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:27.245913 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:28.246484 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.246295 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:28.247085 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.246350 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:28.247085 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:28.246578 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:28.247085 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:28.246647 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:28.526757 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.526641 2582 generic.go:356] "Generic (PLEG): container finished" podID="d5c50b1e-53b1-4dab-87ed-2b246ce43f74" containerID="df4ee074092114caeb4b997a8bc1d440b87b4aaf250598c291cff5a9a1d03c88" exitCode=0 Jul 29 08:18:28.526757 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.526738 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" event={"ID":"d5c50b1e-53b1-4dab-87ed-2b246ce43f74","Type":"ContainerDied","Data":"df4ee074092114caeb4b997a8bc1d440b87b4aaf250598c291cff5a9a1d03c88"} Jul 29 08:18:28.530539 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.530512 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" event={"ID":"4d066258-79b7-4768-b5f2-b11990e7d37e","Type":"ContainerStarted","Data":"dae5a176895a46cfcbe6c09e2ef55ff9a5028004c40ecd905dba32cc6262e925"} Jul 29 08:18:28.530833 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.530810 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:28.530944 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.530883 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:28.530944 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.530930 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:28.547232 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.547207 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:28.547390 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.547338 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:18:28.580244 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:28.580188 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" podStartSLOduration=7.901754191 podStartE2EDuration="27.580173616s" totalImagesPullingTime="19.678419425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:28.579959246 +0000 UTC m=+28.893140100" watchObservedRunningTime="2026-07-29 08:18:28.580173616 +0000 UTC m=+28.893354503" Jul 29 08:18:29.245594 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:29.245553 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:29.245819 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:29.245692 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:29.886323 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:29.886286 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mcpcq"] Jul 29 08:18:29.886735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:29.886456 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:29.886735 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:29.886564 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:29.887087 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:29.887065 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5pfnw"] Jul 29 08:18:29.887183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:29.887166 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:29.887263 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:29.887244 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:29.896993 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:29.896923 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c9tjk"] Jul 29 08:18:29.897079 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:29.897031 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:29.897151 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:29.897124 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:30.537178 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:30.537142 2582 generic.go:356] "Generic (PLEG): container finished" podID="d5c50b1e-53b1-4dab-87ed-2b246ce43f74" containerID="216b41a0bd4ad2fccc84714e32eadf37c38efdc1014dfcb29441f2927123baa9" exitCode=0 Jul 29 08:18:30.537345 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:30.537236 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" event={"ID":"d5c50b1e-53b1-4dab-87ed-2b246ce43f74","Type":"ContainerDied","Data":"216b41a0bd4ad2fccc84714e32eadf37c38efdc1014dfcb29441f2927123baa9"} Jul 29 08:18:31.246045 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:31.246016 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:31.246449 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:31.246022 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:31.246449 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:31.246114 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:31.246449 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:31.246237 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:32.248595 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:32.248438 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:32.248965 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:32.248693 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:32.544598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:32.544567 2582 generic.go:356] "Generic (PLEG): container finished" podID="d5c50b1e-53b1-4dab-87ed-2b246ce43f74" containerID="a9835d4635858bce5610802ac1286060bc5b9d08b57e54c89e0532a1ba36e244" exitCode=0 Jul 29 08:18:32.544793 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:32.544637 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" event={"ID":"d5c50b1e-53b1-4dab-87ed-2b246ce43f74","Type":"ContainerDied","Data":"a9835d4635858bce5610802ac1286060bc5b9d08b57e54c89e0532a1ba36e244"} Jul 29 08:18:33.245563 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:33.245525 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:33.245775 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:33.245579 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:33.245775 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:33.245703 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:33.245890 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:33.245847 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:33.850425 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:33.850388 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:33.850897 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:33.850456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:33.850897 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:33.850539 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:33.850897 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:33.850584 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:33.850897 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:33.850646 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs podName:30908914-7c33-4a7b-832a-bbf63c72b09b nodeName:}" failed. No retries permitted until 2026-07-29 08:19:05.850605369 +0000 UTC m=+66.163786221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs") pod "network-metrics-daemon-5pfnw" (UID: "30908914-7c33-4a7b-832a-bbf63c72b09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:18:33.850897 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:33.850666 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret podName:ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f nodeName:}" failed. No retries permitted until 2026-07-29 08:19:05.850656307 +0000 UTC m=+66.163837159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret") pod "global-pull-secret-syncer-mcpcq" (UID: "ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:18:33.991684 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:33.991655 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:34.051806 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:34.051773 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvsx\" (UniqueName: \"kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx\") pod \"network-check-target-c9tjk\" (UID: \"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd\") " pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:34.052005 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:34.051921 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:18:34.052005 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:34.051977 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:18:34.052005 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:34.051997 2582 projected.go:196] Error preparing data for projected volume kube-api-access-jqvsx for pod openshift-network-diagnostics/network-check-target-c9tjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:34.052172 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:34.052068 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx podName:c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd nodeName:}" failed. No retries permitted until 2026-07-29 08:19:06.052035022 +0000 UTC m=+66.365215863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvsx" (UniqueName: "kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx") pod "network-check-target-c9tjk" (UID: "c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:18:34.052172 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:34.052119 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:34.249229 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:34.249202 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:34.249429 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:34.249328 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mcpcq" podUID="ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f" Jul 29 08:18:35.246176 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:35.246135 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:35.246843 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:35.246260 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9tjk" podUID="c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd" Jul 29 08:18:35.246843 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:35.246726 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:35.246843 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:35.246818 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pfnw" podUID="30908914-7c33-4a7b-832a-bbf63c72b09b" Jul 29 08:18:35.997253 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:35.997225 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-90.ec2.internal" event="NodeReady" Jul 29 08:18:35.997436 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:35.997355 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 08:18:36.045771 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.045715 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:36.052857 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.052826 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g2cp2"] Jul 29 08:18:36.055804 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.055775 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g2cp2" Jul 29 08:18:36.058445 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.058416 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-88dsw\"" Jul 29 08:18:36.058445 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.058419 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 08:18:36.058649 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.058477 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 08:18:36.058877 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.058860 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 08:18:36.059441 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.059421 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kn46w"] Jul 29 08:18:36.061648 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.061611 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.063546 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.063527 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 08:18:36.063898 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.063878 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 08:18:36.064298 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.064279 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-brkp6\"" Jul 29 08:18:36.067737 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.067717 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g2cp2"] Jul 29 08:18:36.077854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.077780 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kn46w"] Jul 29 08:18:36.124227 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.124190 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:36.165276 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.165238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9d8639-d6b0-463a-bd27-e373d9e10b30-config-volume\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.165462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.165286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.165462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.165308 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9d8639-d6b0-463a-bd27-e373d9e10b30-tmp-dir\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.165462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.165338 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gvm2\" (UniqueName: \"kubernetes.io/projected/8a970a63-ce77-4e38-8bbf-3c2a727e0b2f-kube-api-access-5gvm2\") pod \"ingress-canary-g2cp2\" (UID: \"8a970a63-ce77-4e38-8bbf-3c2a727e0b2f\") " pod="openshift-ingress-canary/ingress-canary-g2cp2" Jul 29 08:18:36.165462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.165369 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2wk\" (UniqueName: \"kubernetes.io/projected/ba9d8639-d6b0-463a-bd27-e373d9e10b30-kube-api-access-vm2wk\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.165462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.165393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a970a63-ce77-4e38-8bbf-3c2a727e0b2f-cert\") pod \"ingress-canary-g2cp2\" (UID: \"8a970a63-ce77-4e38-8bbf-3c2a727e0b2f\") " pod="openshift-ingress-canary/ingress-canary-g2cp2" Jul 29 08:18:36.248746 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.248676 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:18:36.251553 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.251530 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 08:18:36.265707 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.265677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2wk\" (UniqueName: \"kubernetes.io/projected/ba9d8639-d6b0-463a-bd27-e373d9e10b30-kube-api-access-vm2wk\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.265851 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.265716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a970a63-ce77-4e38-8bbf-3c2a727e0b2f-cert\") pod \"ingress-canary-g2cp2\" (UID: \"8a970a63-ce77-4e38-8bbf-3c2a727e0b2f\") " pod="openshift-ingress-canary/ingress-canary-g2cp2" Jul 29 08:18:36.265851 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.265793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9d8639-d6b0-463a-bd27-e373d9e10b30-config-volume\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.265851 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.265830 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.265851 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.265849 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9d8639-d6b0-463a-bd27-e373d9e10b30-tmp-dir\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.266051 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:36.265978 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:18:36.266051 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.265974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gvm2\" (UniqueName: \"kubernetes.io/projected/8a970a63-ce77-4e38-8bbf-3c2a727e0b2f-kube-api-access-5gvm2\") pod \"ingress-canary-g2cp2\" (UID: \"8a970a63-ce77-4e38-8bbf-3c2a727e0b2f\") " pod="openshift-ingress-canary/ingress-canary-g2cp2" Jul 29 08:18:36.266135 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:36.266051 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls podName:ba9d8639-d6b0-463a-bd27-e373d9e10b30 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:36.76601997 +0000 UTC m=+37.079200810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls") pod "dns-default-kn46w" (UID: "ba9d8639-d6b0-463a-bd27-e373d9e10b30") : secret "dns-default-metrics-tls" not found Jul 29 08:18:36.266194 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.266182 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9d8639-d6b0-463a-bd27-e373d9e10b30-tmp-dir\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.266401 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.266384 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9d8639-d6b0-463a-bd27-e373d9e10b30-config-volume\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.270268 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.270240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a970a63-ce77-4e38-8bbf-3c2a727e0b2f-cert\") pod \"ingress-canary-g2cp2\" (UID: \"8a970a63-ce77-4e38-8bbf-3c2a727e0b2f\") " pod="openshift-ingress-canary/ingress-canary-g2cp2" Jul 29 08:18:36.277527 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.277494 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2wk\" (UniqueName: \"kubernetes.io/projected/ba9d8639-d6b0-463a-bd27-e373d9e10b30-kube-api-access-vm2wk\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.277646 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.277555 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gvm2\" (UniqueName: \"kubernetes.io/projected/8a970a63-ce77-4e38-8bbf-3c2a727e0b2f-kube-api-access-5gvm2\") pod \"ingress-canary-g2cp2\" (UID: \"8a970a63-ce77-4e38-8bbf-3c2a727e0b2f\") " pod="openshift-ingress-canary/ingress-canary-g2cp2" Jul 29 08:18:36.369656 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.369607 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g2cp2" Jul 29 08:18:36.553735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.553485 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g2cp2"] Jul 29 08:18:36.557962 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.557932 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" event={"ID":"d5c50b1e-53b1-4dab-87ed-2b246ce43f74","Type":"ContainerStarted","Data":"680f6d1822deace02de81aa1e447e3bcad550a4b65e85561ccaba062c69370f1"} Jul 29 08:18:36.558682 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:36.558655 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a970a63_ce77_4e38_8bbf_3c2a727e0b2f.slice/crio-d5b250b7591aa68511fa2b1742b94555a08c74a9f3339d5e30c531f5c1d081c8 WatchSource:0}: Error finding container d5b250b7591aa68511fa2b1742b94555a08c74a9f3339d5e30c531f5c1d081c8: Status 404 returned error can't find the container with id d5b250b7591aa68511fa2b1742b94555a08c74a9f3339d5e30c531f5c1d081c8 Jul 29 08:18:36.770303 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:36.770201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:36.770428 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:36.770329 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:18:36.770428 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:36.770399 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls podName:ba9d8639-d6b0-463a-bd27-e373d9e10b30 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:37.770381949 +0000 UTC m=+38.083562787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls") pod "dns-default-kn46w" (UID: "ba9d8639-d6b0-463a-bd27-e373d9e10b30") : secret "dns-default-metrics-tls" not found Jul 29 08:18:37.245943 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.245905 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:18:37.246114 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.245909 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:18:37.254059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.253653 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 08:18:37.254059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.253705 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b56dd\"" Jul 29 08:18:37.255058 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.254686 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hb4cj\"" Jul 29 08:18:37.255058 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.254878 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 08:18:37.259641 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.259606 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 08:18:37.564171 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.564084 2582 generic.go:356] "Generic (PLEG): container finished" podID="d5c50b1e-53b1-4dab-87ed-2b246ce43f74" containerID="680f6d1822deace02de81aa1e447e3bcad550a4b65e85561ccaba062c69370f1" exitCode=0 Jul 29 08:18:37.564339 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.564182 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" event={"ID":"d5c50b1e-53b1-4dab-87ed-2b246ce43f74","Type":"ContainerDied","Data":"680f6d1822deace02de81aa1e447e3bcad550a4b65e85561ccaba062c69370f1"} Jul 29 08:18:37.565496 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.565473 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g2cp2" event={"ID":"8a970a63-ce77-4e38-8bbf-3c2a727e0b2f","Type":"ContainerStarted","Data":"d5b250b7591aa68511fa2b1742b94555a08c74a9f3339d5e30c531f5c1d081c8"} Jul 29 08:18:37.777839 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:37.777807 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:37.778032 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:37.777927 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:18:37.778032 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:37.777994 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls podName:ba9d8639-d6b0-463a-bd27-e373d9e10b30 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:39.777981482 +0000 UTC m=+40.091162320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls") pod "dns-default-kn46w" (UID: "ba9d8639-d6b0-463a-bd27-e373d9e10b30") : secret "dns-default-metrics-tls" not found Jul 29 08:18:38.571539 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:38.571505 2582 generic.go:356] "Generic (PLEG): container finished" podID="d5c50b1e-53b1-4dab-87ed-2b246ce43f74" containerID="34e3abea8b49b93f2c0a02ff6b9b1fdd874dbe31aa02e78111ad10cbdc4c7b5c" exitCode=0 Jul 29 08:18:38.572034 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:38.571589 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" event={"ID":"d5c50b1e-53b1-4dab-87ed-2b246ce43f74","Type":"ContainerDied","Data":"34e3abea8b49b93f2c0a02ff6b9b1fdd874dbe31aa02e78111ad10cbdc4c7b5c"} Jul 29 08:18:39.577097 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:39.576889 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" event={"ID":"d5c50b1e-53b1-4dab-87ed-2b246ce43f74","Type":"ContainerStarted","Data":"56d5de2f029ae9c97d57ebff9764e2904577803cf1881ea4e5b92d3ff54338e0"} Jul 29 08:18:39.578324 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:39.578301 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g2cp2" event={"ID":"8a970a63-ce77-4e38-8bbf-3c2a727e0b2f","Type":"ContainerStarted","Data":"1a837f4cd3620030127edd725559c64e32a6c6bd77634dfdd083f7aa139a2c1e"} Jul 29 08:18:39.607892 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:39.607842 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j5cxr" podStartSLOduration=8.627832605 podStartE2EDuration="38.607829131s" totalImagesPullingTime="29.979996526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:39.606834772 +0000 UTC m=+39.920015628" watchObservedRunningTime="2026-07-29 08:18:39.607829131 +0000 UTC m=+39.921009990" Jul 29 08:18:39.750414 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:39.750385 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:39.794459 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:39.794423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:39.794564 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:39.794546 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:18:39.794615 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:39.794605 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls podName:ba9d8639-d6b0-463a-bd27-e373d9e10b30 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:43.794588213 +0000 UTC m=+44.107769054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls") pod "dns-default-kn46w" (UID: "ba9d8639-d6b0-463a-bd27-e373d9e10b30") : secret "dns-default-metrics-tls" not found Jul 29 08:18:39.847787 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:39.847756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:39.918939 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:39.918910 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:41.408303 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:41.408262 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:41.508974 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:41.508942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:43.822340 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:43.822298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:43.822764 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:43.822476 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:18:43.822764 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:43.822547 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls podName:ba9d8639-d6b0-463a-bd27-e373d9e10b30 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:51.822531062 +0000 UTC m=+52.135711901 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls") pod "dns-default-kn46w" (UID: "ba9d8639-d6b0-463a-bd27-e373d9e10b30") : secret "dns-default-metrics-tls" not found Jul 29 08:18:50.599823 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.599765 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g2cp2" podStartSLOduration=12.156417173 podStartE2EDuration="14.599746515s" totalImagesPullingTime="2.443329342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:39.634063713 +0000 UTC m=+39.947244573" watchObservedRunningTime="2026-07-29 08:18:50.599746515 +0000 UTC m=+50.912927382" Jul 29 08:18:50.600276 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.599922 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh"] Jul 29 08:18:50.615591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.615559 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-849c4d5579-zcssz"] Jul 29 08:18:50.615746 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.615713 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.618873 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.618847 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 29 08:18:50.618988 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.618871 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 29 08:18:50.619581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.619564 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:18:50.619964 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.619949 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 29 08:18:50.620019 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.619985 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rkghs\"" Jul 29 08:18:50.629428 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.629400 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh"] Jul 29 08:18:50.629541 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.629524 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.631466 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.631442 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-849c4d5579-zcssz"] Jul 29 08:18:50.633189 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.633165 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 29 08:18:50.633189 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.633180 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 29 08:18:50.633344 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.633202 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 08:18:50.633344 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.633169 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 29 08:18:50.633344 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.633260 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tbvxw\"" Jul 29 08:18:50.633344 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.633173 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 29 08:18:50.633524 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.633488 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 08:18:50.725695 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.725659 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8"] Jul 29 08:18:50.746483 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.746441 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:50.747649 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.747603 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8"] Jul 29 08:18:50.749898 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.749877 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 08:18:50.750054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.749909 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jrp2v\"" Jul 29 08:18:50.750054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.749960 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 29 08:18:50.750189 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.750128 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 08:18:50.750283 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.750266 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 29 08:18:50.771612 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.771585 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-stats-auth\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.771740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.771641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c589bb2-2f4a-448e-a03b-2d1579ce3f62-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh\" (UID: \"4c589bb2-2f4a-448e-a03b-2d1579ce3f62\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.771740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.771701 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gx6k\" (UniqueName: \"kubernetes.io/projected/4c589bb2-2f4a-448e-a03b-2d1579ce3f62-kube-api-access-9gx6k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh\" (UID: \"4c589bb2-2f4a-448e-a03b-2d1579ce3f62\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.771740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.771722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.771840 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.771745 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.771840 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.771765 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c589bb2-2f4a-448e-a03b-2d1579ce3f62-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh\" (UID: \"4c589bb2-2f4a-448e-a03b-2d1579ce3f62\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.771840 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.771781 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdsw\" (UniqueName: \"kubernetes.io/projected/d95ac6b4-144f-44f3-841e-80421f1f7559-kube-api-access-vxdsw\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.771927 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.771867 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-default-certificate\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.796723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.796699 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk"] Jul 29 08:18:50.818380 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.818351 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc"] Jul 29 08:18:50.818503 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.818470 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:50.821325 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.821294 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 29 08:18:50.821462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.821329 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9n9bl\"" Jul 29 08:18:50.821462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.821365 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 29 08:18:50.821462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.821302 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 29 08:18:50.821652 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.821299 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:18:50.834516 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.834488 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk"] Jul 29 08:18:50.834650 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.834527 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc"] Jul 29 08:18:50.834650 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.834607 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:18:50.837109 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.837087 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:18:50.837218 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.837150 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 29 08:18:50.837422 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.837406 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 29 08:18:50.837479 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.837465 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-78p6s\"" Jul 29 08:18:50.844171 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.844153 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:50.872370 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872288 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-default-certificate\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.872370 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872328 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:50.872370 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872354 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7mz\" (UniqueName: \"kubernetes.io/projected/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-kube-api-access-db7mz\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:50.872713 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-stats-auth\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.872713 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c589bb2-2f4a-448e-a03b-2d1579ce3f62-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh\" (UID: \"4c589bb2-2f4a-448e-a03b-2d1579ce3f62\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.872713 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gx6k\" (UniqueName: \"kubernetes.io/projected/4c589bb2-2f4a-448e-a03b-2d1579ce3f62-kube-api-access-9gx6k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh\" (UID: \"4c589bb2-2f4a-448e-a03b-2d1579ce3f62\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.872713 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.872713 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:50.872584 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 08:18:50.872713 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872653 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:50.872713 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:50.872671 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:51.372651494 +0000 UTC m=+51.685832349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : secret "router-metrics-certs-default" not found Jul 29 08:18:50.872713 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872710 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.873046 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872745 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c589bb2-2f4a-448e-a03b-2d1579ce3f62-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh\" (UID: \"4c589bb2-2f4a-448e-a03b-2d1579ce3f62\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.873046 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.872769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdsw\" (UniqueName: \"kubernetes.io/projected/d95ac6b4-144f-44f3-841e-80421f1f7559-kube-api-access-vxdsw\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.873046 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:50.872849 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:51.372836914 +0000 UTC m=+51.686017774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : configmap references non-existent config key: service-ca.crt Jul 29 08:18:50.873308 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.873290 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c589bb2-2f4a-448e-a03b-2d1579ce3f62-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh\" (UID: \"4c589bb2-2f4a-448e-a03b-2d1579ce3f62\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.876361 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.876339 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c589bb2-2f4a-448e-a03b-2d1579ce3f62-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh\" (UID: \"4c589bb2-2f4a-448e-a03b-2d1579ce3f62\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.876536 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.876518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-default-certificate\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.876599 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.876581 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-stats-auth\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.883351 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.883333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdsw\" (UniqueName: \"kubernetes.io/projected/d95ac6b4-144f-44f3-841e-80421f1f7559-kube-api-access-vxdsw\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:50.883478 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.883458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gx6k\" (UniqueName: \"kubernetes.io/projected/4c589bb2-2f4a-448e-a03b-2d1579ce3f62-kube-api-access-9gx6k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh\" (UID: \"4c589bb2-2f4a-448e-a03b-2d1579ce3f62\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.924790 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.924754 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" Jul 29 08:18:50.946281 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.946245 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:50.973749 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.973720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xjz\" (UniqueName: \"kubernetes.io/projected/c22b782b-e8c7-40a0-863a-07f3ef1976f3-kube-api-access-c9xjz\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:18:50.973890 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.973788 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:50.973890 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.973811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7mz\" (UniqueName: \"kubernetes.io/projected/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-kube-api-access-db7mz\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:50.973890 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.973831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgc2r\" (UniqueName: \"kubernetes.io/projected/d99f40c8-58ca-4ff1-99ea-c18951b03e15-kube-api-access-fgc2r\") pod \"service-ca-operator-5d6c75f84f-jqgwk\" (UID: \"d99f40c8-58ca-4ff1-99ea-c18951b03e15\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:50.973890 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.973866 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:18:50.974100 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.973908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:50.974100 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.973951 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d99f40c8-58ca-4ff1-99ea-c18951b03e15-serving-cert\") pod \"service-ca-operator-5d6c75f84f-jqgwk\" (UID: \"d99f40c8-58ca-4ff1-99ea-c18951b03e15\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:50.974100 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:50.973987 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 08:18:50.974100 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.974019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d99f40c8-58ca-4ff1-99ea-c18951b03e15-config\") pod \"service-ca-operator-5d6c75f84f-jqgwk\" (UID: \"d99f40c8-58ca-4ff1-99ea-c18951b03e15\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:50.974100 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:50.974032 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls podName:1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec nodeName:}" failed. No retries permitted until 2026-07-29 08:18:51.474018041 +0000 UTC m=+51.787198879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-b2lj8" (UID: "1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec") : secret "cluster-monitoring-operator-tls" not found Jul 29 08:18:50.974510 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.974490 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:50.984650 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:50.984593 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7mz\" (UniqueName: \"kubernetes.io/projected/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-kube-api-access-db7mz\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:51.074708 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.074672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xjz\" (UniqueName: \"kubernetes.io/projected/c22b782b-e8c7-40a0-863a-07f3ef1976f3-kube-api-access-c9xjz\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:18:51.074862 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.074754 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgc2r\" (UniqueName: \"kubernetes.io/projected/d99f40c8-58ca-4ff1-99ea-c18951b03e15-kube-api-access-fgc2r\") pod \"service-ca-operator-5d6c75f84f-jqgwk\" (UID: \"d99f40c8-58ca-4ff1-99ea-c18951b03e15\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:51.074862 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.074788 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:18:51.074862 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.074845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d99f40c8-58ca-4ff1-99ea-c18951b03e15-serving-cert\") pod \"service-ca-operator-5d6c75f84f-jqgwk\" (UID: \"d99f40c8-58ca-4ff1-99ea-c18951b03e15\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:51.074974 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.074874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d99f40c8-58ca-4ff1-99ea-c18951b03e15-config\") pod \"service-ca-operator-5d6c75f84f-jqgwk\" (UID: \"d99f40c8-58ca-4ff1-99ea-c18951b03e15\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:51.074974 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.074880 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 08:18:51.074974 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.074966 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls podName:c22b782b-e8c7-40a0-863a-07f3ef1976f3 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:51.574946709 +0000 UTC m=+51.888127556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls") pod "cluster-samples-operator-6748489656-qv8pc" (UID: "c22b782b-e8c7-40a0-863a-07f3ef1976f3") : secret "samples-operator-tls" not found Jul 29 08:18:51.075509 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.075492 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d99f40c8-58ca-4ff1-99ea-c18951b03e15-config\") pod \"service-ca-operator-5d6c75f84f-jqgwk\" (UID: \"d99f40c8-58ca-4ff1-99ea-c18951b03e15\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:51.077129 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.077113 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d99f40c8-58ca-4ff1-99ea-c18951b03e15-serving-cert\") pod \"service-ca-operator-5d6c75f84f-jqgwk\" (UID: \"d99f40c8-58ca-4ff1-99ea-c18951b03e15\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:51.079203 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.079141 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh"] Jul 29 08:18:51.083256 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.083234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgc2r\" (UniqueName: \"kubernetes.io/projected/d99f40c8-58ca-4ff1-99ea-c18951b03e15-kube-api-access-fgc2r\") pod \"service-ca-operator-5d6c75f84f-jqgwk\" (UID: \"d99f40c8-58ca-4ff1-99ea-c18951b03e15\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:51.083362 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:51.083257 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c589bb2_2f4a_448e_a03b_2d1579ce3f62.slice/crio-0d38f6494fea196284f284df11e7636c5e7bb43e04f06c304eef62b3a7b12691 WatchSource:0}: Error finding container 0d38f6494fea196284f284df11e7636c5e7bb43e04f06c304eef62b3a7b12691: Status 404 returned error can't find the container with id 0d38f6494fea196284f284df11e7636c5e7bb43e04f06c304eef62b3a7b12691 Jul 29 08:18:51.083458 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.083440 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xjz\" (UniqueName: \"kubernetes.io/projected/c22b782b-e8c7-40a0-863a-07f3ef1976f3-kube-api-access-c9xjz\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:18:51.131758 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.131663 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" Jul 29 08:18:51.282933 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.282901 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk"] Jul 29 08:18:51.286386 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:51.286360 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99f40c8_58ca_4ff1_99ea_c18951b03e15.slice/crio-4cf69813bae66d472bb40cb01c5f9e029d86df5a62545de99f164a54691c32f8 WatchSource:0}: Error finding container 4cf69813bae66d472bb40cb01c5f9e029d86df5a62545de99f164a54691c32f8: Status 404 returned error can't find the container with id 4cf69813bae66d472bb40cb01c5f9e029d86df5a62545de99f164a54691c32f8 Jul 29 08:18:51.378432 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.378396 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:51.378651 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.378447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:51.378651 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.378563 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 08:18:51.378651 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.378580 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:52.378562892 +0000 UTC m=+52.691743735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : configmap references non-existent config key: service-ca.crt Jul 29 08:18:51.378651 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.378641 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:52.378604151 +0000 UTC m=+52.691784989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : secret "router-metrics-certs-default" not found Jul 29 08:18:51.479417 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.479384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:51.479588 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.479543 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 08:18:51.479657 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.479614 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls podName:1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec nodeName:}" failed. No retries permitted until 2026-07-29 08:18:52.479598105 +0000 UTC m=+52.792778943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-b2lj8" (UID: "1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec") : secret "cluster-monitoring-operator-tls" not found Jul 29 08:18:51.580875 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.580841 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:18:51.581067 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.581005 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 08:18:51.581139 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.581084 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls podName:c22b782b-e8c7-40a0-863a-07f3ef1976f3 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:52.581064878 +0000 UTC m=+52.894245716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls") pod "cluster-samples-operator-6748489656-qv8pc" (UID: "c22b782b-e8c7-40a0-863a-07f3ef1976f3") : secret "samples-operator-tls" not found Jul 29 08:18:51.605461 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.605421 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" event={"ID":"d99f40c8-58ca-4ff1-99ea-c18951b03e15","Type":"ContainerStarted","Data":"4cf69813bae66d472bb40cb01c5f9e029d86df5a62545de99f164a54691c32f8"} Jul 29 08:18:51.606486 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.606461 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" event={"ID":"4c589bb2-2f4a-448e-a03b-2d1579ce3f62","Type":"ContainerStarted","Data":"0d38f6494fea196284f284df11e7636c5e7bb43e04f06c304eef62b3a7b12691"} Jul 29 08:18:51.884594 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:51.884504 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:18:51.884764 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.884654 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:18:51.884764 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:51.884721 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls podName:ba9d8639-d6b0-463a-bd27-e373d9e10b30 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:07.884702637 +0000 UTC m=+68.197883476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls") pod "dns-default-kn46w" (UID: "ba9d8639-d6b0-463a-bd27-e373d9e10b30") : secret "dns-default-metrics-tls" not found Jul 29 08:18:52.388541 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.388313 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:52.388541 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.388374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:52.388541 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:52.388470 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 08:18:52.388541 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:52.388539 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:54.388516116 +0000 UTC m=+54.701696961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : secret "router-metrics-certs-default" not found Jul 29 08:18:52.389029 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:52.388567 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:54.388547279 +0000 UTC m=+54.701728118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : configmap references non-existent config key: service-ca.crt Jul 29 08:18:52.489021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.488982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:52.489218 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:52.489153 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 08:18:52.489271 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:52.489226 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls podName:1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec nodeName:}" failed. No retries permitted until 2026-07-29 08:18:54.489206881 +0000 UTC m=+54.802387719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-b2lj8" (UID: "1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec") : secret "cluster-monitoring-operator-tls" not found Jul 29 08:18:52.538675 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.538642 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q"] Jul 29 08:18:52.559513 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.559485 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q"] Jul 29 08:18:52.559680 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.559585 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:18:52.561935 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.561900 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 29 08:18:52.562057 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.561971 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2kl6x\"" Jul 29 08:18:52.562057 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.561979 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 29 08:18:52.590205 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.590171 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:18:52.590205 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.590230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:18:52.590509 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.590287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/33011701-b1db-40f6-879e-76e769b46374-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:18:52.590509 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:52.590344 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 08:18:52.590509 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:52.590417 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls podName:c22b782b-e8c7-40a0-863a-07f3ef1976f3 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:54.590392592 +0000 UTC m=+54.903573437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls") pod "cluster-samples-operator-6748489656-qv8pc" (UID: "c22b782b-e8c7-40a0-863a-07f3ef1976f3") : secret "samples-operator-tls" not found Jul 29 08:18:52.691233 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.691154 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:18:52.691643 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.691246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/33011701-b1db-40f6-879e-76e769b46374-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:18:52.691643 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:52.691305 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 08:18:52.691643 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:52.691376 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert podName:33011701-b1db-40f6-879e-76e769b46374 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:53.191357434 +0000 UTC m=+53.504538292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5fc4q" (UID: "33011701-b1db-40f6-879e-76e769b46374") : secret "networking-console-plugin-cert" not found Jul 29 08:18:52.691948 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:52.691927 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/33011701-b1db-40f6-879e-76e769b46374-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:18:53.159388 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:53.159350 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:53.194949 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:53.194915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:18:53.195087 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:53.195030 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 08:18:53.195121 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:53.195088 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert podName:33011701-b1db-40f6-879e-76e769b46374 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:54.195071238 +0000 UTC m=+54.508252079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5fc4q" (UID: "33011701-b1db-40f6-879e-76e769b46374") : secret "networking-console-plugin-cert" not found Jul 29 08:18:53.261205 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:53.261171 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:54.201600 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:54.201574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:18:54.201916 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:54.201724 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 08:18:54.201916 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:54.201783 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert podName:33011701-b1db-40f6-879e-76e769b46374 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:56.201766438 +0000 UTC m=+56.514947277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5fc4q" (UID: "33011701-b1db-40f6-879e-76e769b46374") : secret "networking-console-plugin-cert" not found Jul 29 08:18:54.403309 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:54.403224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:54.403309 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:54.403279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:54.403590 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:54.403378 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 08:18:54.403590 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:54.403446 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:58.403429722 +0000 UTC m=+58.716610564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : secret "router-metrics-certs-default" not found Jul 29 08:18:54.403590 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:54.403481 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:58.403458994 +0000 UTC m=+58.716639838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : configmap references non-existent config key: service-ca.crt Jul 29 08:18:54.504095 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:54.504054 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:54.504269 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:54.504205 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 08:18:54.504269 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:54.504265 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls podName:1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec nodeName:}" failed. No retries permitted until 2026-07-29 08:18:58.504251007 +0000 UTC m=+58.817431846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-b2lj8" (UID: "1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec") : secret "cluster-monitoring-operator-tls" not found Jul 29 08:18:54.604437 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:54.604399 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:18:54.604579 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:54.604556 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 08:18:54.604657 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:54.604632 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls podName:c22b782b-e8c7-40a0-863a-07f3ef1976f3 nodeName:}" failed. No retries permitted until 2026-07-29 08:18:58.604602194 +0000 UTC m=+58.917783038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls") pod "cluster-samples-operator-6748489656-qv8pc" (UID: "c22b782b-e8c7-40a0-863a-07f3ef1976f3") : secret "samples-operator-tls" not found Jul 29 08:18:54.616055 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:54.616021 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" event={"ID":"d99f40c8-58ca-4ff1-99ea-c18951b03e15","Type":"ContainerStarted","Data":"ff6a6b8e5c0526f3b7d8c36047115459316f4c9d55fc82ecdd5161caeebb596e"} Jul 29 08:18:54.617701 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:54.617679 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" event={"ID":"4c589bb2-2f4a-448e-a03b-2d1579ce3f62","Type":"ContainerStarted","Data":"a2e70a81bd3c7c13084e1cf3054b8ca5fcefc5632eadcb0c54fac5334277d16b"} Jul 29 08:18:54.634086 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:54.634045 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" podStartSLOduration=1.766015618 podStartE2EDuration="4.634033038s" totalImagesPullingTime="2.86801742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:54.633290866 +0000 UTC m=+54.946471727" watchObservedRunningTime="2026-07-29 08:18:54.634033038 +0000 UTC m=+54.947213897" Jul 29 08:18:54.657024 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:54.656914 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" podStartSLOduration=1.591241982 podStartE2EDuration="4.65689296s" totalImagesPullingTime="3.065650978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:18:54.656709353 +0000 UTC m=+54.969890213" watchObservedRunningTime="2026-07-29 08:18:54.65689296 +0000 UTC m=+54.970073821" Jul 29 08:18:55.449945 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.449911 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-86bbc"] Jul 29 08:18:55.453090 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.453066 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.459304 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.459283 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 08:18:55.459416 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.459317 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 08:18:55.459698 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.459682 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 08:18:55.459764 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.459706 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 08:18:55.460109 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.460094 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rbl5j\"" Jul 29 08:18:55.507237 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.507198 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-86bbc"] Jul 29 08:18:55.512004 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.511975 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.512168 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.512072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.512168 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.512107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-data-volume\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.512168 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.512146 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjsn\" (UniqueName: \"kubernetes.io/projected/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-api-access-nfjsn\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.512307 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.512238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-crio-socket\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.597827 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.597796 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:55.613103 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.613075 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-data-volume\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.613259 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.613131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjsn\" (UniqueName: \"kubernetes.io/projected/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-api-access-nfjsn\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.613259 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.613241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-crio-socket\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.613368 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.613321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.613416 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.613375 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-crio-socket\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.613416 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.613391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.613513 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.613437 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-data-volume\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.613513 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:55.613479 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:18:55.613513 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:55.613485 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:18:55.613682 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:55.613542 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:18:56.113524792 +0000 UTC m=+56.426705634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : configmap "kube-rbac-proxy" not found Jul 29 08:18:55.613682 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:55.613559 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:18:56.113549624 +0000 UTC m=+56.426730469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : secret "insights-runtime-extractor-tls" not found Jul 29 08:18:55.641972 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.641934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjsn\" (UniqueName: \"kubernetes.io/projected/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-api-access-nfjsn\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:55.740459 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.740424 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:55.886834 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.886797 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:55.984381 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:55.984347 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:56.117339 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.117241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:56.117518 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.117345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:56.117518 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:56.117461 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:18:56.117642 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:56.117544 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:18:57.117521864 +0000 UTC m=+57.430702720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : configmap "kube-rbac-proxy" not found Jul 29 08:18:56.118017 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:56.117997 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:18:56.118100 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:56.118084 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:18:57.118067322 +0000 UTC m=+57.431248162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : secret "insights-runtime-extractor-tls" not found Jul 29 08:18:56.218366 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.218324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:18:56.218558 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:56.218469 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 08:18:56.218558 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:56.218534 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert podName:33011701-b1db-40f6-879e-76e769b46374 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:00.218519474 +0000 UTC m=+60.531700312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5fc4q" (UID: "33011701-b1db-40f6-879e-76e769b46374") : secret "networking-console-plugin-cert" not found Jul 29 08:18:56.373228 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.373138 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:56.472223 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.472188 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:56.596392 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.596361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:56.720959 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.720922 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:56.947692 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.947664 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:56.987483 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.987396 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj"] Jul 29 08:18:56.990053 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.990026 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj" Jul 29 08:18:56.992786 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.992757 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 29 08:18:56.996426 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.996387 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 29 08:18:56.996920 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:56.996897 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-thlxz\"" Jul 29 08:18:57.015562 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.015532 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj"] Jul 29 08:18:57.025830 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.025793 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqklk\" (UniqueName: \"kubernetes.io/projected/ac99683b-c447-476c-af04-eee88294cbc1-kube-api-access-xqklk\") pod \"migrator-6865c9967-p6pgj\" (UID: \"ac99683b-c447-476c-af04-eee88294cbc1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj" Jul 29 08:18:57.126929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.126890 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:57.127146 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.126976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:57.127146 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.127041 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqklk\" (UniqueName: \"kubernetes.io/projected/ac99683b-c447-476c-af04-eee88294cbc1-kube-api-access-xqklk\") pod \"migrator-6865c9967-p6pgj\" (UID: \"ac99683b-c447-476c-af04-eee88294cbc1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj" Jul 29 08:18:57.127146 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:57.127077 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:18:57.127146 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:57.127090 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:18:57.127146 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:57.127149 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:18:59.127127922 +0000 UTC m=+59.440308778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : secret "insights-runtime-extractor-tls" not found Jul 29 08:18:57.127338 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:57.127169 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:18:59.127158983 +0000 UTC m=+59.440339822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : configmap "kube-rbac-proxy" not found Jul 29 08:18:57.155225 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.155191 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqklk\" (UniqueName: \"kubernetes.io/projected/ac99683b-c447-476c-af04-eee88294cbc1-kube-api-access-xqklk\") pod \"migrator-6865c9967-p6pgj\" (UID: \"ac99683b-c447-476c-af04-eee88294cbc1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj" Jul 29 08:18:57.165976 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.165949 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:57.291504 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.291411 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:57.303254 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.303230 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj" Jul 29 08:18:57.408361 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.408336 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cwbns_79567eb3-7610-4e72-b625-6700f758a78a/dns-node-resolver/0.log" Jul 29 08:18:57.439364 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.439331 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj"] Jul 29 08:18:57.442174 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:18:57.442147 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac99683b_c447_476c_af04_eee88294cbc1.slice/crio-7ffc35e8d5733535db8e1c61c161b2995751766a70a57be925d9a43eceb6ec77 WatchSource:0}: Error finding container 7ffc35e8d5733535db8e1c61c161b2995751766a70a57be925d9a43eceb6ec77: Status 404 returned error can't find the container with id 7ffc35e8d5733535db8e1c61c161b2995751766a70a57be925d9a43eceb6ec77 Jul 29 08:18:57.626968 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:57.626876 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj" event={"ID":"ac99683b-c447-476c-af04-eee88294cbc1","Type":"ContainerStarted","Data":"7ffc35e8d5733535db8e1c61c161b2995751766a70a57be925d9a43eceb6ec77"} Jul 29 08:18:58.216633 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:58.216594 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5vj99_8971e266-0ba5-4a9f-a516-83954d7c971a/node-ca/0.log" Jul 29 08:18:58.274699 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:58.274663 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:58.377911 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:58.377876 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:18:58.438155 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:58.438114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:58.438155 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:58.438159 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:18:58.438405 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:58.438264 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 08:18:58.438405 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:58.438320 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:06.438303903 +0000 UTC m=+66.751484740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : secret "router-metrics-certs-default" not found Jul 29 08:18:58.438405 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:58.438348 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:06.438342069 +0000 UTC m=+66.751522907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : configmap references non-existent config key: service-ca.crt Jul 29 08:18:58.539744 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:58.539659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:18:58.539899 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:58.539812 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 08:18:58.539961 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:58.539898 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls podName:1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec nodeName:}" failed. No retries permitted until 2026-07-29 08:19:06.539875892 +0000 UTC m=+66.853056735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-b2lj8" (UID: "1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec") : secret "cluster-monitoring-operator-tls" not found Jul 29 08:18:58.640455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:58.640423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:18:58.640858 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:58.640581 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 08:18:58.640858 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:58.640659 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls podName:c22b782b-e8c7-40a0-863a-07f3ef1976f3 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:06.6406411 +0000 UTC m=+66.953821938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls") pod "cluster-samples-operator-6748489656-qv8pc" (UID: "c22b782b-e8c7-40a0-863a-07f3ef1976f3") : secret "samples-operator-tls" not found Jul 29 08:18:59.019287 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:59.019250 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g2cp2_8a970a63-ce77-4e38-8bbf-3c2a727e0b2f/serve-healthcheck-canary/0.log" Jul 29 08:18:59.144609 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:59.144576 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:59.144808 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:18:59.144671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:18:59.144808 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:59.144778 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:18:59.144808 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:59.144786 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:18:59.144931 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:59.144855 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:19:03.144834674 +0000 UTC m=+63.458015516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : secret "insights-runtime-extractor-tls" not found Jul 29 08:18:59.144931 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:18:59.144874 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:19:03.144864623 +0000 UTC m=+63.458045467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : configmap "kube-rbac-proxy" not found Jul 29 08:19:00.091333 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:00.091302 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:00.255151 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:00.255124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:19:00.255290 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:00.255271 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 08:19:00.255343 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:00.255333 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert podName:33011701-b1db-40f6-879e-76e769b46374 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:08.255317091 +0000 UTC m=+68.568497928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5fc4q" (UID: "33011701-b1db-40f6-879e-76e769b46374") : secret "networking-console-plugin-cert" not found Jul 29 08:19:00.552187 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:00.552151 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sx2cm" Jul 29 08:19:00.635981 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:00.635936 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj" event={"ID":"ac99683b-c447-476c-af04-eee88294cbc1","Type":"ContainerStarted","Data":"4d6bfec63ef06913becb688ff51dd72d7028b0a4a92bd6cb2d522d94b7a038e7"} Jul 29 08:19:00.635981 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:00.635988 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj" event={"ID":"ac99683b-c447-476c-af04-eee88294cbc1","Type":"ContainerStarted","Data":"146e0f91cc6c20a74819d8b3716e9f5328f4170625d29978651b44b02ae7eb28"} Jul 29 08:19:00.672664 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:00.672590 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-p6pgj" podStartSLOduration=2.002724494 podStartE2EDuration="4.672573464s" totalImagesPullingTime="2.66984897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:00.67199072 +0000 UTC m=+60.985171580" watchObservedRunningTime="2026-07-29 08:19:00.672573464 +0000 UTC m=+60.985754303" Jul 29 08:19:01.950367 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:01.950307 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:02.078880 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:02.078847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:02.459561 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:02.459526 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:02.554857 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:02.554824 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:02.799139 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:02.799060 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:02.950253 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:02.950218 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:03.183116 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:03.183028 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:19:03.183116 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:03.183092 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:19:03.183529 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:03.183187 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:19:03.183529 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:03.183253 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:19:11.183235706 +0000 UTC m=+71.496416547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : secret "insights-runtime-extractor-tls" not found Jul 29 08:19:03.183529 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:03.183191 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:19:03.183529 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:03.183326 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:19:11.183314546 +0000 UTC m=+71.496495383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : configmap "kube-rbac-proxy" not found Jul 29 08:19:03.828958 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:03.828926 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:04.017568 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:04.017496 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:04.567951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:04.567919 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:04.673274 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:04.673238 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:05.098266 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:05.098231 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:05.678279 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:05.678238 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:05.909002 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:05.908973 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:19:05.909160 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:05.909017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:19:05.911913 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:05.911897 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 08:19:05.912051 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:05.912035 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 08:19:05.919856 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:05.919835 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 08:19:05.919965 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:05.919899 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs podName:30908914-7c33-4a7b-832a-bbf63c72b09b nodeName:}" failed. No retries permitted until 2026-07-29 08:20:09.919881533 +0000 UTC m=+130.233062370 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs") pod "network-metrics-daemon-5pfnw" (UID: "30908914-7c33-4a7b-832a-bbf63c72b09b") : secret "metrics-daemon-secret" not found Jul 29 08:19:05.922159 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:05.922138 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f-original-pull-secret\") pod \"global-pull-secret-syncer-mcpcq\" (UID: \"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f\") " pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:19:05.954482 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:05.954388 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:05.958408 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:05.958393 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mcpcq" Jul 29 08:19:06.041994 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.041967 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:06.100516 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.100485 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mcpcq"] Jul 29 08:19:06.104113 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:06.104084 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbd8f37_c168_4e7a_b06a_9bfdd9579c7f.slice/crio-eb390a1944b3832dfea94192f85d309114bef3fdda30b739c4987c677fa9ff90 WatchSource:0}: Error finding container eb390a1944b3832dfea94192f85d309114bef3fdda30b739c4987c677fa9ff90: Status 404 returned error can't find the container with id eb390a1944b3832dfea94192f85d309114bef3fdda30b739c4987c677fa9ff90 Jul 29 08:19:06.111584 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.111561 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvsx\" (UniqueName: \"kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx\") pod \"network-check-target-c9tjk\" (UID: \"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd\") " pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:19:06.125507 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.125477 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 08:19:06.134704 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.134686 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 08:19:06.145169 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.145137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvsx\" (UniqueName: \"kubernetes.io/projected/c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd-kube-api-access-jqvsx\") pod \"network-check-target-c9tjk\" (UID: \"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd\") " pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:19:06.233103 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.233077 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:06.360819 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.360783 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b56dd\"" Jul 29 08:19:06.369076 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.369050 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:19:06.409136 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.409067 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:06.476914 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.476874 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:06.496158 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.496133 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c9tjk"] Jul 29 08:19:06.498509 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:06.498482 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a2ea3c_92cd_45a6_a55e_38ab4b60f2dd.slice/crio-8af7ce0264ed8806fe3525c496f7d289b651c54823f6ebed4a52cdf3ca050810 WatchSource:0}: Error finding container 8af7ce0264ed8806fe3525c496f7d289b651c54823f6ebed4a52cdf3ca050810: Status 404 returned error can't find the container with id 8af7ce0264ed8806fe3525c496f7d289b651c54823f6ebed4a52cdf3ca050810 Jul 29 08:19:06.516067 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.516027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:06.516259 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.516084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:06.516259 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:06.516197 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 08:19:06.516375 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:06.516272 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:22.516252673 +0000 UTC m=+82.829433512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : secret "router-metrics-certs-default" not found Jul 29 08:19:06.516375 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:06.516288 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle podName:d95ac6b4-144f-44f3-841e-80421f1f7559 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:22.516281349 +0000 UTC m=+82.829462187 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle") pod "router-default-849c4d5579-zcssz" (UID: "d95ac6b4-144f-44f3-841e-80421f1f7559") : configmap references non-existent config key: service-ca.crt Jul 29 08:19:06.617403 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.617367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:19:06.617615 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:06.617551 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 08:19:06.617715 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:06.617649 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls podName:1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec nodeName:}" failed. No retries permitted until 2026-07-29 08:19:22.617613348 +0000 UTC m=+82.930794205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-b2lj8" (UID: "1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec") : secret "cluster-monitoring-operator-tls" not found Jul 29 08:19:06.651201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.651160 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mcpcq" event={"ID":"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f","Type":"ContainerStarted","Data":"eb390a1944b3832dfea94192f85d309114bef3fdda30b739c4987c677fa9ff90"} Jul 29 08:19:06.652436 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.652401 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c9tjk" event={"ID":"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd","Type":"ContainerStarted","Data":"8af7ce0264ed8806fe3525c496f7d289b651c54823f6ebed4a52cdf3ca050810"} Jul 29 08:19:06.718593 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.718557 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:19:06.721698 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.721667 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c22b782b-e8c7-40a0-863a-07f3ef1976f3-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-qv8pc\" (UID: \"c22b782b-e8c7-40a0-863a-07f3ef1976f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:19:06.748052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.747979 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-78p6s\"" Jul 29 08:19:06.752827 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.752801 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" Jul 29 08:19:06.805521 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.805469 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:06.918977 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.918942 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc"] Jul 29 08:19:06.994167 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:06.994140 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:07.099529 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:07.099437 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:07.381978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:07.381889 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:07.657223 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:07.657083 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" event={"ID":"c22b782b-e8c7-40a0-863a-07f3ef1976f3","Type":"ContainerStarted","Data":"0ef3138262bcb21d509122c8ec187464871f89e5acdd4277bdd2794c28ee592f"} Jul 29 08:19:07.930634 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:07.930531 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:19:07.938724 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:07.938650 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba9d8639-d6b0-463a-bd27-e373d9e10b30-metrics-tls\") pod \"dns-default-kn46w\" (UID: \"ba9d8639-d6b0-463a-bd27-e373d9e10b30\") " pod="openshift-dns/dns-default-kn46w" Jul 29 08:19:08.182924 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:08.182654 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-brkp6\"" Jul 29 08:19:08.186738 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:08.186709 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kn46w" Jul 29 08:19:08.251052 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:08.251017 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:08.334664 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:08.334601 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:19:08.334818 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:08.334802 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 08:19:08.334902 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:08.334890 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert podName:33011701-b1db-40f6-879e-76e769b46374 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:24.334868354 +0000 UTC m=+84.648049208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5fc4q" (UID: "33011701-b1db-40f6-879e-76e769b46374") : secret "networking-console-plugin-cert" not found Jul 29 08:19:08.434146 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:08.434059 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kn46w"] Jul 29 08:19:08.440065 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:08.440024 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9d8639_d6b0_463a_bd27_e373d9e10b30.slice/crio-fc1d3b6381a418e0b1d5db313b56732ce4d473b74d4346db836c5e25a4915f9c WatchSource:0}: Error finding container fc1d3b6381a418e0b1d5db313b56732ce4d473b74d4346db836c5e25a4915f9c: Status 404 returned error can't find the container with id fc1d3b6381a418e0b1d5db313b56732ce4d473b74d4346db836c5e25a4915f9c Jul 29 08:19:08.636081 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:08.636003 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:08.661663 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:08.661597 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kn46w" event={"ID":"ba9d8639-d6b0-463a-bd27-e373d9e10b30","Type":"ContainerStarted","Data":"fc1d3b6381a418e0b1d5db313b56732ce4d473b74d4346db836c5e25a4915f9c"} Jul 29 08:19:08.734499 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:08.734470 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:08.848465 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:08.848426 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:09.036912 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:09.036828 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:09.391989 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:09.391906 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:10.185108 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:10.185072 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:10.578846 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:10.578811 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:10.778709 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:10.778672 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:10.968390 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:10.968352 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:11.185396 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:11.185362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:11.265753 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:11.265671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:19:11.265919 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:11.265816 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:19:11.265985 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:11.265936 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:19:11.266038 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:11.266007 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls podName:f2eec8d1-92dc-4480-8bd0-df9135ee6f8a nodeName:}" failed. No retries permitted until 2026-07-29 08:19:27.265988301 +0000 UTC m=+87.579169159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-86bbc" (UID: "f2eec8d1-92dc-4480-8bd0-df9135ee6f8a") : secret "insights-runtime-extractor-tls" not found Jul 29 08:19:11.266320 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:11.266295 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:19:12.009160 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.009125 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:12.386496 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.386410 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:12.602387 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.602350 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:12.677158 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.677072 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" event={"ID":"c22b782b-e8c7-40a0-863a-07f3ef1976f3","Type":"ContainerStarted","Data":"a9acc761d963d68c3cf893bdaecbf3af2313aecb7517c3c32919b4f973f54ce9"} Jul 29 08:19:12.677158 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.677134 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" event={"ID":"c22b782b-e8c7-40a0-863a-07f3ef1976f3","Type":"ContainerStarted","Data":"97861ccb756d7b0f87c0d2d02789282c4af9bf1f48a3bb6865ca47cf07cb8e9d"} Jul 29 08:19:12.678501 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.678476 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c9tjk" event={"ID":"c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd","Type":"ContainerStarted","Data":"15e734ac299994293825fe93342bebd34f64eb67b56ad1509ae31211bb33421f"} Jul 29 08:19:12.678675 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.678543 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:19:12.679810 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.679788 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mcpcq" event={"ID":"ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f","Type":"ContainerStarted","Data":"c73b27c3f55b0508bee324c1615567e89433259c21aa4024cf51f079f8662ea2"} Jul 29 08:19:12.695758 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.695691 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-qv8pc" podStartSLOduration=18.069684348 podStartE2EDuration="22.695677253s" totalImagesPullingTime="4.625992905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:12.694780396 +0000 UTC m=+73.007961256" watchObservedRunningTime="2026-07-29 08:19:12.695677253 +0000 UTC m=+73.008858137" Jul 29 08:19:12.712562 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.712509 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c9tjk" podStartSLOduration=66.611411769 podStartE2EDuration="1m11.712492196s" totalImagesPullingTime="5.101080427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:12.710903503 +0000 UTC m=+73.024084365" watchObservedRunningTime="2026-07-29 08:19:12.712492196 +0000 UTC m=+73.025673058" Jul 29 08:19:12.732072 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.732001 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mcpcq" podStartSLOduration=66.225108348 podStartE2EDuration="1m11.731981442s" totalImagesPullingTime="5.506873094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:12.731708553 +0000 UTC m=+73.044889426" watchObservedRunningTime="2026-07-29 08:19:12.731981442 +0000 UTC m=+73.045162303" Jul 29 08:19:12.775700 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:12.775644 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:13.685311 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:13.685280 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kn46w" event={"ID":"ba9d8639-d6b0-463a-bd27-e373d9e10b30","Type":"ContainerStarted","Data":"29eabefc1ead4f3342368bc46a8b25e5465ddf492f27523feed338ebdcca23b8"} Jul 29 08:19:13.685798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:13.685332 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kn46w" event={"ID":"ba9d8639-d6b0-463a-bd27-e373d9e10b30","Type":"ContainerStarted","Data":"e11111a27b27a3fc0a093f80ad67861ce758a8dff62996911184b5a07d62ef6f"} Jul 29 08:19:13.685798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:13.685610 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kn46w" Jul 29 08:19:13.717478 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:13.717427 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kn46w" podStartSLOduration=32.669004481 podStartE2EDuration="37.717410997s" totalImagesPullingTime="5.048406516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:13.717000296 +0000 UTC m=+74.030181185" watchObservedRunningTime="2026-07-29 08:19:13.717410997 +0000 UTC m=+74.030591856" Jul 29 08:19:16.950882 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:16.950837 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:17.223490 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:17.223454 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:17.374149 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:17.374117 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:19.079854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:19.079816 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:19.158152 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:19.158118 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:22.558579 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.558464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:22.558579 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.558500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:22.559112 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.559094 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95ac6b4-144f-44f3-841e-80421f1f7559-service-ca-bundle\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:22.560954 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.560934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95ac6b4-144f-44f3-841e-80421f1f7559-metrics-certs\") pod \"router-default-849c4d5579-zcssz\" (UID: \"d95ac6b4-144f-44f3-841e-80421f1f7559\") " pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:22.659567 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.659523 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:19:22.662055 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.662035 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-b2lj8\" (UID: \"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:19:22.741735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.741694 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tbvxw\"" Jul 29 08:19:22.749867 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.749842 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:22.859339 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.859304 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jrp2v\"" Jul 29 08:19:22.867551 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.867525 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" Jul 29 08:19:22.883197 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:22.883169 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-849c4d5579-zcssz"] Jul 29 08:19:22.886016 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:22.885988 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95ac6b4_144f_44f3_841e_80421f1f7559.slice/crio-e449a5cd4759a5b37df0022d8a814925e825a8edd3097510343e680a2fd31023 WatchSource:0}: Error finding container e449a5cd4759a5b37df0022d8a814925e825a8edd3097510343e680a2fd31023: Status 404 returned error can't find the container with id e449a5cd4759a5b37df0022d8a814925e825a8edd3097510343e680a2fd31023 Jul 29 08:19:23.018083 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:23.018023 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8"] Jul 29 08:19:23.022396 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:23.022370 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aed42ac_58e1_4a5c_b9fb_7acc1f0c94ec.slice/crio-23079e5201167d645861950696ab69c07658803b9c02346d5778478dde68985b WatchSource:0}: Error finding container 23079e5201167d645861950696ab69c07658803b9c02346d5778478dde68985b: Status 404 returned error can't find the container with id 23079e5201167d645861950696ab69c07658803b9c02346d5778478dde68985b Jul 29 08:19:23.691127 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:23.691100 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kn46w" Jul 29 08:19:23.719996 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:23.719903 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-849c4d5579-zcssz" event={"ID":"d95ac6b4-144f-44f3-841e-80421f1f7559","Type":"ContainerStarted","Data":"923fef4759de2a08d000129b93eed15e75c3ab140efb2cc4e06712791ec5b821"} Jul 29 08:19:23.719996 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:23.719958 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-849c4d5579-zcssz" event={"ID":"d95ac6b4-144f-44f3-841e-80421f1f7559","Type":"ContainerStarted","Data":"e449a5cd4759a5b37df0022d8a814925e825a8edd3097510343e680a2fd31023"} Jul 29 08:19:23.723645 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:23.722777 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" event={"ID":"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec","Type":"ContainerStarted","Data":"23079e5201167d645861950696ab69c07658803b9c02346d5778478dde68985b"} Jul 29 08:19:23.750766 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:23.750737 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:23.753654 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:23.753608 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:23.775316 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:23.775268 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-849c4d5579-zcssz" podStartSLOduration=33.775250648 podStartE2EDuration="33.775250648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:23.739644351 +0000 UTC m=+84.052825205" watchObservedRunningTime="2026-07-29 08:19:23.775250648 +0000 UTC m=+84.088431507" Jul 29 08:19:24.373868 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:24.373829 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:19:24.376486 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:24.376460 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33011701-b1db-40f6-879e-76e769b46374-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5fc4q\" (UID: \"33011701-b1db-40f6-879e-76e769b46374\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:19:24.673422 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:24.673332 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2kl6x\"" Jul 29 08:19:24.681477 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:24.681448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" Jul 29 08:19:24.727030 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:24.726997 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:24.728505 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:24.728482 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-849c4d5579-zcssz" Jul 29 08:19:25.117317 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.117287 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q"] Jul 29 08:19:25.121150 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:25.121120 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33011701_b1db_40f6_879e_76e769b46374.slice/crio-9c76f56def46ce223e8548a486e1f56ca9b3522d7518d4df470c9bfc827b6a95 WatchSource:0}: Error finding container 9c76f56def46ce223e8548a486e1f56ca9b3522d7518d4df470c9bfc827b6a95: Status 404 returned error can't find the container with id 9c76f56def46ce223e8548a486e1f56ca9b3522d7518d4df470c9bfc827b6a95 Jul 29 08:19:25.355257 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.355170 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:25.439411 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.439374 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:25.501188 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.501147 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd"] Jul 29 08:19:25.506121 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.505212 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd" Jul 29 08:19:25.507934 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.507909 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lctrz\"" Jul 29 08:19:25.508072 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.508015 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 29 08:19:25.510787 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.510763 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd"] Jul 29 08:19:25.521203 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.521180 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:25.584945 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.584915 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f5d8e82e-0f59-4605-8151-3aeaee89fe3e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5dnwd\" (UID: \"f5d8e82e-0f59-4605-8151-3aeaee89fe3e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd" Jul 29 08:19:25.667309 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.667221 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:25.686015 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.685980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f5d8e82e-0f59-4605-8151-3aeaee89fe3e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5dnwd\" (UID: \"f5d8e82e-0f59-4605-8151-3aeaee89fe3e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd" Jul 29 08:19:25.688487 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.688458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f5d8e82e-0f59-4605-8151-3aeaee89fe3e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5dnwd\" (UID: \"f5d8e82e-0f59-4605-8151-3aeaee89fe3e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd" Jul 29 08:19:25.731823 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.731785 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" event={"ID":"1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec","Type":"ContainerStarted","Data":"3d064806bb7276a9845185b42deb9980e1a96daf57eb1612786d1d05a25fe254"} Jul 29 08:19:25.732931 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.732910 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" event={"ID":"33011701-b1db-40f6-879e-76e769b46374","Type":"ContainerStarted","Data":"9c76f56def46ce223e8548a486e1f56ca9b3522d7518d4df470c9bfc827b6a95"} Jul 29 08:19:25.747551 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.747506 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-b2lj8" podStartSLOduration=33.751000047 podStartE2EDuration="35.747491251s" totalImagesPullingTime="1.996491204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:25.747326694 +0000 UTC m=+86.060507555" watchObservedRunningTime="2026-07-29 08:19:25.747491251 +0000 UTC m=+86.060672110" Jul 29 08:19:25.816654 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.816600 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd" Jul 29 08:19:25.963168 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:25.963135 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd"] Jul 29 08:19:25.966886 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:25.966859 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d8e82e_0f59_4605_8151_3aeaee89fe3e.slice/crio-597ee21038cf2e35e75ab03352e74870c9273649e35bfe327c99695bb090faea WatchSource:0}: Error finding container 597ee21038cf2e35e75ab03352e74870c9273649e35bfe327c99695bb090faea: Status 404 returned error can't find the container with id 597ee21038cf2e35e75ab03352e74870c9273649e35bfe327c99695bb090faea Jul 29 08:19:26.737246 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:26.737210 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd" event={"ID":"f5d8e82e-0f59-4605-8151-3aeaee89fe3e","Type":"ContainerStarted","Data":"597ee21038cf2e35e75ab03352e74870c9273649e35bfe327c99695bb090faea"} Jul 29 08:19:27.299539 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.299494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:19:27.302178 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.302153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2eec8d1-92dc-4480-8bd0-df9135ee6f8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbc\" (UID: \"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a\") " pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:19:27.564787 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.564695 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rbl5j\"" Jul 29 08:19:27.572701 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.572680 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-86bbc" Jul 29 08:19:27.721866 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.721835 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-86bbc"] Jul 29 08:19:27.724684 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:27.724649 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2eec8d1_92dc_4480_8bd0_df9135ee6f8a.slice/crio-adcaec8eab01805e15b5946bec2760a54137e76d3f3f99cf5bc341efe6518775 WatchSource:0}: Error finding container adcaec8eab01805e15b5946bec2760a54137e76d3f3f99cf5bc341efe6518775: Status 404 returned error can't find the container with id adcaec8eab01805e15b5946bec2760a54137e76d3f3f99cf5bc341efe6518775 Jul 29 08:19:27.742272 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.742239 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" event={"ID":"33011701-b1db-40f6-879e-76e769b46374","Type":"ContainerStarted","Data":"336ac667b4db1bc792be2c85bcaa54d0213340b72b0ea3faea0a8e73f1adaf2f"} Jul 29 08:19:27.743492 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.743469 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86bbc" event={"ID":"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a","Type":"ContainerStarted","Data":"adcaec8eab01805e15b5946bec2760a54137e76d3f3f99cf5bc341efe6518775"} Jul 29 08:19:27.744922 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.744899 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd" event={"ID":"f5d8e82e-0f59-4605-8151-3aeaee89fe3e","Type":"ContainerStarted","Data":"e327dbfcb2eb7613a95e2374bb2edb61b01e42c13595ef1b57b1a5cf2b6a651d"} Jul 29 08:19:27.745094 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.745077 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd" Jul 29 08:19:27.749502 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.749485 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd" Jul 29 08:19:27.756821 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.756783 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5fc4q" podStartSLOduration=34.090431422 podStartE2EDuration="35.756768456s" totalImagesPullingTime="1.666337034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:27.75596569 +0000 UTC m=+88.069146541" watchObservedRunningTime="2026-07-29 08:19:27.756768456 +0000 UTC m=+88.069949621" Jul 29 08:19:27.768912 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.768862 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5dnwd" podStartSLOduration=1.362709731 podStartE2EDuration="2.768848726s" totalImagesPullingTime="1.406138995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:27.768460636 +0000 UTC m=+88.081641495" watchObservedRunningTime="2026-07-29 08:19:27.768848726 +0000 UTC m=+88.082029588" Jul 29 08:19:27.859324 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.859231 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:27.961521 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:27.961487 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:28.552944 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:28.552798 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:28.620913 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:28.620878 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:29.302789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:29.302756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:32.766745 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:32.766711 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86bbc" event={"ID":"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a","Type":"ContainerStarted","Data":"74839e41139b47c0d67238093b447de4b9036dc816f5a5f678dc99a06b14e2b7"} Jul 29 08:19:32.831457 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:32.831425 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:32.950302 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:32.950219 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:34.050449 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:34.050418 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:34.105467 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:34.105430 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:34.426352 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:34.426264 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:35.004755 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:35.004717 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:35.072156 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:35.072123 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:35.277171 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:35.277100 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:35.332152 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:35.332120 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:36.782971 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:36.782941 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86bbc" event={"ID":"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a","Type":"ContainerStarted","Data":"74ced746dcf21aa0923c7ce3f19d30ddf7c679b06774f8361c5a061ccb4fd5b5"} Jul 29 08:19:37.681219 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:37.681183 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:37.970863 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:37.970831 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:38.045008 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.044977 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg"] Jul 29 08:19:38.047814 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.047787 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.053868 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.053845 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 29 08:19:38.054169 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.053851 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 29 08:19:38.054329 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.054311 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kl5pl"] Jul 29 08:19:38.054816 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.054613 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vnr7f\"" Jul 29 08:19:38.054816 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.054689 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 29 08:19:38.054816 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.054689 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 08:19:38.057193 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.057173 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.060401 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.060287 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 08:19:38.060401 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.060309 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 08:19:38.060401 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.060322 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 08:19:38.060613 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.060438 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wmnl4\"" Jul 29 08:19:38.073872 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.073849 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg"] Jul 29 08:19:38.097930 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.097891 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:38.194192 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194156 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrg4m\" (UniqueName: \"kubernetes.io/projected/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-api-access-rrg4m\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.194422 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3b42f2e-0adf-469f-9e88-befa1af691da-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.194422 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194294 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.194422 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194350 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksbh2\" (UniqueName: \"kubernetes.io/projected/847664e3-1d31-418d-b9d4-85c12c854a24-kube-api-access-ksbh2\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.194422 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.194709 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194475 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-wtmp\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.194709 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194532 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/847664e3-1d31-418d-b9d4-85c12c854a24-metrics-client-ca\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.194709 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194562 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.194709 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/847664e3-1d31-418d-b9d4-85c12c854a24-root\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.194709 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194684 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/847664e3-1d31-418d-b9d4-85c12c854a24-sys\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.194884 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194721 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-textfile\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.194884 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194753 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.194884 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194784 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-accelerators-collector-config\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.194884 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194822 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c3b42f2e-0adf-469f-9e88-befa1af691da-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.194884 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.194848 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-tls\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.295968 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.295888 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/847664e3-1d31-418d-b9d4-85c12c854a24-sys\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.295968 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.295952 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-textfile\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296175 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.295983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296175 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296011 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/847664e3-1d31-418d-b9d4-85c12c854a24-sys\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296175 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-accelerators-collector-config\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296175 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c3b42f2e-0adf-469f-9e88-befa1af691da-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.296175 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-tls\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296175 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrg4m\" (UniqueName: \"kubernetes.io/projected/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-api-access-rrg4m\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.296455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3b42f2e-0adf-469f-9e88-befa1af691da-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.296455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.296455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296293 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksbh2\" (UniqueName: \"kubernetes.io/projected/847664e3-1d31-418d-b9d4-85c12c854a24-kube-api-access-ksbh2\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296317 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.296455 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:38.296367 2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 08:19:38.296714 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:38.296460 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-tls podName:847664e3-1d31-418d-b9d4-85c12c854a24 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:38.796435561 +0000 UTC m=+99.109616402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-tls") pod "node-exporter-kl5pl" (UID: "847664e3-1d31-418d-b9d4-85c12c854a24") : secret "node-exporter-tls" not found Jul 29 08:19:38.296714 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296484 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-wtmp\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296714 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296371 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-wtmp\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296714 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296698 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/847664e3-1d31-418d-b9d4-85c12c854a24-metrics-client-ca\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296908 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296726 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-accelerators-collector-config\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296908 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.296908 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/847664e3-1d31-418d-b9d4-85c12c854a24-root\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296908 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296836 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-textfile\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.296908 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296860 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c3b42f2e-0adf-469f-9e88-befa1af691da-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.296908 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.296891 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/847664e3-1d31-418d-b9d4-85c12c854a24-root\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.297201 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:38.296921 2582 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 29 08:19:38.297201 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:38.296986 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-tls podName:c3b42f2e-0adf-469f-9e88-befa1af691da nodeName:}" failed. No retries permitted until 2026-07-29 08:19:38.796968701 +0000 UTC m=+99.110149552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-v6trg" (UID: "c3b42f2e-0adf-469f-9e88-befa1af691da") : secret "kube-state-metrics-tls" not found Jul 29 08:19:38.297458 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.297422 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3b42f2e-0adf-469f-9e88-befa1af691da-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.297547 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.297507 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.297654 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.297602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/847664e3-1d31-418d-b9d4-85c12c854a24-metrics-client-ca\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.299234 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.299213 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.299820 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.299802 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.304671 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.304599 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksbh2\" (UniqueName: \"kubernetes.io/projected/847664e3-1d31-418d-b9d4-85c12c854a24-kube-api-access-ksbh2\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.304766 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.304727 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrg4m\" (UniqueName: \"kubernetes.io/projected/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-api-access-rrg4m\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.411340 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.411304 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:38.516905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.516871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:38.611650 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.611564 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:38.712305 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.712270 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:38.801782 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.801751 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-tls\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.801960 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.801808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.805293 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.804747 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/847664e3-1d31-418d-b9d4-85c12c854a24-node-exporter-tls\") pod \"node-exporter-kl5pl\" (UID: \"847664e3-1d31-418d-b9d4-85c12c854a24\") " pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.805293 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.804920 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3b42f2e-0adf-469f-9e88-befa1af691da-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-v6trg\" (UID: \"c3b42f2e-0adf-469f-9e88-befa1af691da\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.959459 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.959428 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" Jul 29 08:19:38.969500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.969468 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kl5pl" Jul 29 08:19:38.983572 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:38.983541 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:39.087895 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.087859 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:39.102810 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.102778 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:19:39.106825 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.106801 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.109776 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.109752 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 08:19:39.109916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.109760 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 08:19:39.110249 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.110209 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 08:19:39.110411 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.110369 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 08:19:39.110534 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.110428 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 08:19:39.110534 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.110502 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 08:19:39.110613 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.110598 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 08:19:39.110613 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.110605 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 08:19:39.110753 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.110735 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 08:19:39.110838 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.110823 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v4wpr\"" Jul 29 08:19:39.128044 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.128015 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:19:39.205365 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205329 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205563 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-web-config\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205563 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-volume\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205563 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205563 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205508 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205562 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205607 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-out\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205777 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205982 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205814 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205982 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205847 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205982 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205920 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rd6h\" (UniqueName: \"kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-kube-api-access-7rd6h\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.205982 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.205953 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307319 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-volume\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307319 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307276 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307546 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307546 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307546 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307417 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307546 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-out\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307546 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307483 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307546 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307508 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307546 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307886 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307576 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rd6h\" (UniqueName: \"kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-kube-api-access-7rd6h\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307886 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307606 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307886 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.307886 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.307693 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-web-config\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.308059 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:39.307992 2582 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 29 08:19:39.308059 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:19:39.308051 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-main-tls podName:4f14ba66-4b09-4afa-8993-8d4ed54f40d2 nodeName:}" failed. No retries permitted until 2026-07-29 08:19:39.808031555 +0000 UTC m=+100.121212397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2") : secret "alertmanager-main-tls" not found Jul 29 08:19:39.308518 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.308308 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.309680 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.308816 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.309680 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.309463 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.311375 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.311355 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.311968 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.311921 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.313235 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.313211 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-volume\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.313348 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.313211 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-out\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.313613 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.313593 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-web-config\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.313822 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.313801 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.313906 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.313861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.314585 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.314556 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.318448 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.318426 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rd6h\" (UniqueName: \"kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-kube-api-access-7rd6h\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.414055 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.414016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:39.518115 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.518086 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:39.612535 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.612444 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:39.714082 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.714046 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:39.812227 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.812181 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:39.814478 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.814451 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:39.815390 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:39.815366 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:40.018720 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:40.018681 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:19:40.028979 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:40.028809 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847664e3_1d31_418d_b9d4_85c12c854a24.slice/crio-93020a6aea1eca86eddb2247954207f58e909af728116fcca1fceee9149f1981 WatchSource:0}: Error finding container 93020a6aea1eca86eddb2247954207f58e909af728116fcca1fceee9149f1981: Status 404 returned error can't find the container with id 93020a6aea1eca86eddb2247954207f58e909af728116fcca1fceee9149f1981 Jul 29 08:19:40.188112 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:40.188082 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:19:40.191943 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:40.191912 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f14ba66_4b09_4afa_8993_8d4ed54f40d2.slice/crio-c88ebea03fa028e4a63f9cb2637ace129ca79c55ed47c3247da795da073647e8 WatchSource:0}: Error finding container c88ebea03fa028e4a63f9cb2637ace129ca79c55ed47c3247da795da073647e8: Status 404 returned error can't find the container with id c88ebea03fa028e4a63f9cb2637ace129ca79c55ed47c3247da795da073647e8 Jul 29 08:19:40.192658 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:40.192574 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg"] Jul 29 08:19:40.194671 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:40.194648 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b42f2e_0adf_469f_9e88_befa1af691da.slice/crio-42c9d62d3ee57b43c62bf7e63a837b226f63e3a03d7892b6d8f7637ae0168d25 WatchSource:0}: Error finding container 42c9d62d3ee57b43c62bf7e63a837b226f63e3a03d7892b6d8f7637ae0168d25: Status 404 returned error can't find the container with id 42c9d62d3ee57b43c62bf7e63a837b226f63e3a03d7892b6d8f7637ae0168d25 Jul 29 08:19:40.801809 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:40.801665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" event={"ID":"c3b42f2e-0adf-469f-9e88-befa1af691da","Type":"ContainerStarted","Data":"42c9d62d3ee57b43c62bf7e63a837b226f63e3a03d7892b6d8f7637ae0168d25"} Jul 29 08:19:40.804398 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:40.804350 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86bbc" event={"ID":"f2eec8d1-92dc-4480-8bd0-df9135ee6f8a","Type":"ContainerStarted","Data":"a76749034af463d7864f9c9357472ddebe106740c7a47092e66edb8401f60eed"} Jul 29 08:19:40.806640 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:40.806594 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerStarted","Data":"c88ebea03fa028e4a63f9cb2637ace129ca79c55ed47c3247da795da073647e8"} Jul 29 08:19:40.808088 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:40.808029 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kl5pl" event={"ID":"847664e3-1d31-418d-b9d4-85c12c854a24","Type":"ContainerStarted","Data":"93020a6aea1eca86eddb2247954207f58e909af728116fcca1fceee9149f1981"} Jul 29 08:19:40.825123 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:40.824717 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-86bbc" podStartSLOduration=33.622090182 podStartE2EDuration="45.824697212s" totalImagesPullingTime="12.20260703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:18:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:40.82424708 +0000 UTC m=+101.137427941" watchObservedRunningTime="2026-07-29 08:19:40.824697212 +0000 UTC m=+101.137878075" Jul 29 08:19:40.933368 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:40.933329 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:41.035866 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:41.035759 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:41.182455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:41.182410 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:41.263637 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:41.263589 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:41.814434 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:41.814405 2582 generic.go:356] "Generic (PLEG): container finished" podID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerID="a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89" exitCode=0 Jul 29 08:19:41.814602 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:41.814482 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerDied","Data":"a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89"} Jul 29 08:19:41.816381 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:41.816359 2582 generic.go:356] "Generic (PLEG): container finished" podID="847664e3-1d31-418d-b9d4-85c12c854a24" containerID="8ed2a49edc4966d36b8f89e21d687b17bf99394686429dd31732e26f0ff1aab7" exitCode=0 Jul 29 08:19:41.816493 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:41.816464 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kl5pl" event={"ID":"847664e3-1d31-418d-b9d4-85c12c854a24","Type":"ContainerDied","Data":"8ed2a49edc4966d36b8f89e21d687b17bf99394686429dd31732e26f0ff1aab7"} Jul 29 08:19:41.818783 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:41.818758 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" event={"ID":"c3b42f2e-0adf-469f-9e88-befa1af691da","Type":"ContainerStarted","Data":"87ad90c225c43a83edf3b766c75a1ae070c8c6603c3671065f1082faaac489b3"} Jul 29 08:19:41.852880 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:41.852849 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:41.930153 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:41.930113 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:42.044002 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.043968 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:42.519332 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.519304 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5848859c-49qsj"] Jul 29 08:19:42.521905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.521889 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.525033 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.525015 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 29 08:19:42.525168 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.525151 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 08:19:42.525378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.525361 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 29 08:19:42.525511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.525495 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 29 08:19:42.525596 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.525582 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-adtefuc9o11rv\"" Jul 29 08:19:42.525678 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.525601 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4qphm\"" Jul 29 08:19:42.536054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.535293 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b214eb-198b-4652-aac4-515feb24caef-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.536054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.535376 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/86b214eb-198b-4652-aac4-515feb24caef-metrics-server-audit-profiles\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.536054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.535434 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/86b214eb-198b-4652-aac4-515feb24caef-secret-metrics-server-tls\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.536054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.535488 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/86b214eb-198b-4652-aac4-515feb24caef-client-ca-bundle\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.536054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.535543 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782xr\" (UniqueName: \"kubernetes.io/projected/86b214eb-198b-4652-aac4-515feb24caef-kube-api-access-782xr\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.536054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.535578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/86b214eb-198b-4652-aac4-515feb24caef-audit-log\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.536054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.535612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/86b214eb-198b-4652-aac4-515feb24caef-secret-metrics-server-client-certs\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.540247 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.540213 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5848859c-49qsj"] Jul 29 08:19:42.636426 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.636389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/86b214eb-198b-4652-aac4-515feb24caef-secret-metrics-server-tls\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.636633 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.636451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/86b214eb-198b-4652-aac4-515feb24caef-client-ca-bundle\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.636633 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.636483 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782xr\" (UniqueName: \"kubernetes.io/projected/86b214eb-198b-4652-aac4-515feb24caef-kube-api-access-782xr\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.636633 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.636501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/86b214eb-198b-4652-aac4-515feb24caef-audit-log\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.636633 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.636519 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/86b214eb-198b-4652-aac4-515feb24caef-secret-metrics-server-client-certs\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.636633 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.636547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b214eb-198b-4652-aac4-515feb24caef-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.636633 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.636598 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/86b214eb-198b-4652-aac4-515feb24caef-metrics-server-audit-profiles\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.637065 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.637031 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/86b214eb-198b-4652-aac4-515feb24caef-audit-log\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.637378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.637355 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86b214eb-198b-4652-aac4-515feb24caef-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.637527 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.637510 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/86b214eb-198b-4652-aac4-515feb24caef-metrics-server-audit-profiles\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.639257 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.639226 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/86b214eb-198b-4652-aac4-515feb24caef-secret-metrics-server-client-certs\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.639577 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.639559 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/86b214eb-198b-4652-aac4-515feb24caef-secret-metrics-server-tls\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.639642 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.639591 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/86b214eb-198b-4652-aac4-515feb24caef-client-ca-bundle\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.654862 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.654832 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-782xr\" (UniqueName: \"kubernetes.io/projected/86b214eb-198b-4652-aac4-515feb24caef-kube-api-access-782xr\") pod \"metrics-server-5848859c-49qsj\" (UID: \"86b214eb-198b-4652-aac4-515feb24caef\") " pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.774390 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.774312 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:42.825238 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.825201 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kl5pl" event={"ID":"847664e3-1d31-418d-b9d4-85c12c854a24","Type":"ContainerStarted","Data":"0dfe9ecfe1f28e4a25dbea785904bc1e24a11eac63ee91f9216a7f588072d2b8"} Jul 29 08:19:42.825426 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.825272 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kl5pl" event={"ID":"847664e3-1d31-418d-b9d4-85c12c854a24","Type":"ContainerStarted","Data":"8f05e6831fc134fc3f27d002870106902ebae81b7af1441b2c029cda99b0d467"} Jul 29 08:19:42.827456 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.827428 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" event={"ID":"c3b42f2e-0adf-469f-9e88-befa1af691da","Type":"ContainerStarted","Data":"3034b80c7c51040a18ea50b17d1f5bbbc12a67c3c77fbb6565e2ac9a1ef548dd"} Jul 29 08:19:42.827565 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.827483 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" event={"ID":"c3b42f2e-0adf-469f-9e88-befa1af691da","Type":"ContainerStarted","Data":"397d66c80b7fa98e24d137744709ca981601c3e824aeeecade08f45fea80a0cb"} Jul 29 08:19:42.831854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.831830 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:19:42.870055 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.869988 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kl5pl" podStartSLOduration=3.896625422 podStartE2EDuration="4.869967295s" totalImagesPullingTime="886.482048ms" totalInitContainerRuntime="86.859825ms" isStatefulPod=true podCreationTimestamp="2026-07-29 08:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:42.86896729 +0000 UTC m=+103.182148152" watchObservedRunningTime="2026-07-29 08:19:42.869967295 +0000 UTC m=+103.183148156" Jul 29 08:19:42.912031 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.911975 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v6trg" podStartSLOduration=3.4694157150000002 podStartE2EDuration="4.911959344s" totalImagesPullingTime="1.442543629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:42.910097309 +0000 UTC m=+103.223278170" watchObservedRunningTime="2026-07-29 08:19:42.911959344 +0000 UTC m=+103.225140203" Jul 29 08:19:42.918249 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.918217 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:42.975498 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:42.975468 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5848859c-49qsj"] Jul 29 08:19:42.981264 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:42.981222 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b214eb_198b_4652_aac4_515feb24caef.slice/crio-2731ffe92380867e65a012fb24cd0ebb6cd6e47d15621dab6366cec688567d9a WatchSource:0}: Error finding container 2731ffe92380867e65a012fb24cd0ebb6cd6e47d15621dab6366cec688567d9a: Status 404 returned error can't find the container with id 2731ffe92380867e65a012fb24cd0ebb6cd6e47d15621dab6366cec688567d9a Jul 29 08:19:43.688288 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:43.688260 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c9tjk" Jul 29 08:19:43.834077 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:43.834039 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5848859c-49qsj" event={"ID":"86b214eb-198b-4652-aac4-515feb24caef","Type":"ContainerStarted","Data":"2731ffe92380867e65a012fb24cd0ebb6cd6e47d15621dab6366cec688567d9a"} Jul 29 08:19:44.188764 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:44.188728 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:44.289263 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:44.289230 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:44.842518 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:44.842478 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerStarted","Data":"2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e"} Jul 29 08:19:44.843106 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:44.842570 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerStarted","Data":"a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8"} Jul 29 08:19:44.843106 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:44.842613 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerStarted","Data":"641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7"} Jul 29 08:19:45.868010 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:45.867984 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerStarted","Data":"df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58"} Jul 29 08:19:45.868356 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:45.868079 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerStarted","Data":"1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074"} Jul 29 08:19:46.362181 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:46.362151 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:46.467082 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:46.467052 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:46.876057 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:46.876019 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerStarted","Data":"8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece"} Jul 29 08:19:46.877740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:46.877707 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5848859c-49qsj" event={"ID":"86b214eb-198b-4652-aac4-515feb24caef","Type":"ContainerStarted","Data":"346711c3a3a24d8e634e0d0e69edc7b01340ec78e0ed64aa5b019861741ef165"} Jul 29 08:19:46.906333 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:46.906280 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.517051917 podStartE2EDuration="7.906263947s" totalImagesPullingTime="5.370753286s" totalInitContainerRuntime="18.458744ms" isStatefulPod=false podCreationTimestamp="2026-07-29 08:19:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:46.905900224 +0000 UTC m=+107.219081084" watchObservedRunningTime="2026-07-29 08:19:46.906263947 +0000 UTC m=+107.219444807" Jul 29 08:19:46.931091 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:46.931036 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5848859c-49qsj" podStartSLOduration=2.137745982 podStartE2EDuration="4.931018336s" totalImagesPullingTime="2.793272354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:19:46.92946896 +0000 UTC m=+107.242649823" watchObservedRunningTime="2026-07-29 08:19:46.931018336 +0000 UTC m=+107.244199195" Jul 29 08:19:48.426885 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:48.426844 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:48.526736 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:48.526695 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:49.061859 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:49.061826 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:49.162326 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:49.162297 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:51.259112 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.259080 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-xwpsg"] Jul 29 08:19:51.261452 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.261435 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xwpsg" Jul 29 08:19:51.264113 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.264086 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 08:19:51.264539 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.264523 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 08:19:51.264907 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.264893 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hvsn7\"" Jul 29 08:19:51.269363 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.269345 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:51.278261 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.278241 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xwpsg"] Jul 29 08:19:51.309594 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.309549 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5975t\" (UniqueName: \"kubernetes.io/projected/bf3a040b-6446-4f34-8d64-da906783e51e-kube-api-access-5975t\") pod \"downloads-7c97dc4555-xwpsg\" (UID: \"bf3a040b-6446-4f34-8d64-da906783e51e\") " pod="openshift-console/downloads-7c97dc4555-xwpsg" Jul 29 08:19:51.365397 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.365368 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:51.410840 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.410805 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5975t\" (UniqueName: \"kubernetes.io/projected/bf3a040b-6446-4f34-8d64-da906783e51e-kube-api-access-5975t\") pod \"downloads-7c97dc4555-xwpsg\" (UID: \"bf3a040b-6446-4f34-8d64-da906783e51e\") " pod="openshift-console/downloads-7c97dc4555-xwpsg" Jul 29 08:19:51.420527 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.420503 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5975t\" (UniqueName: \"kubernetes.io/projected/bf3a040b-6446-4f34-8d64-da906783e51e-kube-api-access-5975t\") pod \"downloads-7c97dc4555-xwpsg\" (UID: \"bf3a040b-6446-4f34-8d64-da906783e51e\") " pod="openshift-console/downloads-7c97dc4555-xwpsg" Jul 29 08:19:51.463904 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.463873 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:51.557392 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.557313 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:51.571291 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.571265 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xwpsg" Jul 29 08:19:51.706603 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.706575 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xwpsg"] Jul 29 08:19:51.710517 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:51.710488 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3a040b_6446_4f34_8d64_da906783e51e.slice/crio-3c05aae9a39206269b1193b51d7ef02a5a0fdd5a0e91e235f337ce18e3d1ecc4 WatchSource:0}: Error finding container 3c05aae9a39206269b1193b51d7ef02a5a0fdd5a0e91e235f337ce18e3d1ecc4: Status 404 returned error can't find the container with id 3c05aae9a39206269b1193b51d7ef02a5a0fdd5a0e91e235f337ce18e3d1ecc4 Jul 29 08:19:51.738039 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.738005 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:51.899859 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.899781 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xwpsg" event={"ID":"bf3a040b-6446-4f34-8d64-da906783e51e","Type":"ContainerStarted","Data":"3c05aae9a39206269b1193b51d7ef02a5a0fdd5a0e91e235f337ce18e3d1ecc4"} Jul 29 08:19:51.962498 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:51.962466 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:52.063395 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:52.063360 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:52.165669 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:52.165571 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:52.263333 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:52.263292 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:52.365128 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:52.365093 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:52.950399 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:52.950364 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:53.150378 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:53.150344 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:53.550516 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:53.550482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:54.258267 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:54.258236 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:54.357308 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:54.357267 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:54.459274 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:54.459236 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:19:57.030131 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.030093 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b78c9495-pbbhc"] Jul 29 08:19:57.034427 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.034399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.037429 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.037408 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 08:19:57.038090 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.038058 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 08:19:57.038224 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.038174 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 08:19:57.038591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.038570 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p8ccp\"" Jul 29 08:19:57.038591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.038586 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 08:19:57.038766 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.038638 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 08:19:57.052568 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.052545 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b78c9495-pbbhc"] Jul 29 08:19:57.065782 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.065672 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctjw\" (UniqueName: \"kubernetes.io/projected/37ceb931-70a0-44ae-86f1-b978d171bd31-kube-api-access-bctjw\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.065782 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.065730 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-oauth-serving-cert\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.066065 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.065944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-console-config\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.066065 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.065979 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-service-ca\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.066065 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.066033 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-serving-cert\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.066065 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.066059 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-oauth-config\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.167010 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.166971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-serving-cert\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.167010 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.167007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-oauth-config\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.167251 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.167070 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctjw\" (UniqueName: \"kubernetes.io/projected/37ceb931-70a0-44ae-86f1-b978d171bd31-kube-api-access-bctjw\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.167251 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.167087 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-oauth-serving-cert\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.167251 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.167118 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-console-config\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.167251 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.167144 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-service-ca\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.167924 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.167902 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-console-config\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.168047 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.167936 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-oauth-serving-cert\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.168047 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.167936 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-service-ca\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.169876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.169855 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-oauth-config\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.169969 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.169939 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-serving-cert\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.176602 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.176547 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctjw\" (UniqueName: \"kubernetes.io/projected/37ceb931-70a0-44ae-86f1-b978d171bd31-kube-api-access-bctjw\") pod \"console-7b78c9495-pbbhc\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.346187 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.346081 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:19:57.499156 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.499128 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b78c9495-pbbhc"] Jul 29 08:19:57.501884 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:19:57.501853 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ceb931_70a0_44ae_86f1_b978d171bd31.slice/crio-ab2c65f0eedd8ab26bb0aaaa74b9c7d1f05074fa9577eec7a304a22b9be0eef9 WatchSource:0}: Error finding container ab2c65f0eedd8ab26bb0aaaa74b9c7d1f05074fa9577eec7a304a22b9be0eef9: Status 404 returned error can't find the container with id ab2c65f0eedd8ab26bb0aaaa74b9c7d1f05074fa9577eec7a304a22b9be0eef9 Jul 29 08:19:57.924049 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:19:57.924014 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b78c9495-pbbhc" event={"ID":"37ceb931-70a0-44ae-86f1-b978d171bd31","Type":"ContainerStarted","Data":"ab2c65f0eedd8ab26bb0aaaa74b9c7d1f05074fa9577eec7a304a22b9be0eef9"} Jul 29 08:20:00.940560 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:00.940467 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b78c9495-pbbhc" event={"ID":"37ceb931-70a0-44ae-86f1-b978d171bd31","Type":"ContainerStarted","Data":"11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9"} Jul 29 08:20:00.958819 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:00.958756 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b78c9495-pbbhc" podStartSLOduration=0.789475794 podStartE2EDuration="3.958734712s" totalImagesPullingTime="3.169258918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:20:00.95787474 +0000 UTC m=+121.271055601" watchObservedRunningTime="2026-07-29 08:20:00.958734712 +0000 UTC m=+121.271915572" Jul 29 08:20:02.832929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:02.832896 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:20:02.833404 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:02.832960 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:20:04.074876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:04.074839 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:04.174529 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:04.174495 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:07.346525 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.346468 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:20:07.347032 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.346546 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:20:07.352101 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.352076 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:20:07.362081 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.362054 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f545848f-cjcpw"] Jul 29 08:20:07.478540 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.478494 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f545848f-cjcpw"] Jul 29 08:20:07.478707 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.478646 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.487577 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.487398 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 08:20:07.566190 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.566153 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-trusted-ca-bundle\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.566388 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.566216 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqrg\" (UniqueName: \"kubernetes.io/projected/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-kube-api-access-8sqrg\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.566388 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.566363 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-service-ca\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.566508 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.566417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-oauth-serving-cert\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.566508 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.566473 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-serving-cert\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.566615 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.566512 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-config\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.566615 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.566535 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-oauth-config\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.570123 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.570096 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:07.635505 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.635408 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:07.667598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.667542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-service-ca\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.667809 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.667603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-oauth-serving-cert\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.667809 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.667661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-serving-cert\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.667809 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.667694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-config\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.667809 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.667720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-oauth-config\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.667809 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.667767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-trusted-ca-bundle\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.667809 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.667803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqrg\" (UniqueName: \"kubernetes.io/projected/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-kube-api-access-8sqrg\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.668387 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.668364 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-service-ca\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.668492 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.668422 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-config\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.668639 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.668602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-oauth-serving-cert\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.668707 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.668691 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-trusted-ca-bundle\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.671194 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.671167 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-serving-cert\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.671299 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.671175 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-oauth-config\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.677503 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.677480 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqrg\" (UniqueName: \"kubernetes.io/projected/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-kube-api-access-8sqrg\") pod \"console-6f545848f-cjcpw\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.790922 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.790886 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:07.977402 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:07.977371 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:20:09.994192 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:09.994157 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:20:09.996879 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:09.996847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30908914-7c33-4a7b-832a-bbf63c72b09b-metrics-certs\") pod \"network-metrics-daemon-5pfnw\" (UID: \"30908914-7c33-4a7b-832a-bbf63c72b09b\") " pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:20:10.267492 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.267464 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hb4cj\"" Jul 29 08:20:10.275969 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.275940 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pfnw" Jul 29 08:20:10.316417 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.316388 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f545848f-cjcpw"] Jul 29 08:20:10.320169 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:20:10.320140 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b28c8fe_71fb_4242_8caa_cff2ee11dd3a.slice/crio-1e93cc38738af23d2b5bf829cc824848972d072ea10d1b13d4c648f55c797511 WatchSource:0}: Error finding container 1e93cc38738af23d2b5bf829cc824848972d072ea10d1b13d4c648f55c797511: Status 404 returned error can't find the container with id 1e93cc38738af23d2b5bf829cc824848972d072ea10d1b13d4c648f55c797511 Jul 29 08:20:10.434591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.434567 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5pfnw"] Jul 29 08:20:10.468347 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:20:10.468311 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30908914_7c33_4a7b_832a_bbf63c72b09b.slice/crio-9eb57d8a746867a33719b35ceaa96fcf18348623c2b68caa4cc5be1435b78d6b WatchSource:0}: Error finding container 9eb57d8a746867a33719b35ceaa96fcf18348623c2b68caa4cc5be1435b78d6b: Status 404 returned error can't find the container with id 9eb57d8a746867a33719b35ceaa96fcf18348623c2b68caa4cc5be1435b78d6b Jul 29 08:20:10.987465 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.987028 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xwpsg" event={"ID":"bf3a040b-6446-4f34-8d64-da906783e51e","Type":"ContainerStarted","Data":"45fb98e9d153880dcbf3dd3fdac62254e1dd0a1c45c94014c193c7564310f5cf"} Jul 29 08:20:10.987688 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.987665 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-xwpsg" Jul 29 08:20:10.990589 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.989392 2582 patch_prober.go:28] interesting pod/downloads-7c97dc4555-xwpsg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.20:8080/\": dial tcp 10.134.0.20:8080: connect: connection refused" start-of-body= Jul 29 08:20:10.990589 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.989435 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-xwpsg" podUID="bf3a040b-6446-4f34-8d64-da906783e51e" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.20:8080/\": dial tcp 10.134.0.20:8080: connect: connection refused" Jul 29 08:20:10.990589 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.989847 2582 generic.go:356] "Generic (PLEG): container finished" podID="d99f40c8-58ca-4ff1-99ea-c18951b03e15" containerID="ff6a6b8e5c0526f3b7d8c36047115459316f4c9d55fc82ecdd5161caeebb596e" exitCode=0 Jul 29 08:20:10.990589 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.989938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" event={"ID":"d99f40c8-58ca-4ff1-99ea-c18951b03e15","Type":"ContainerDied","Data":"ff6a6b8e5c0526f3b7d8c36047115459316f4c9d55fc82ecdd5161caeebb596e"} Jul 29 08:20:10.990589 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.990273 2582 scope.go:117] "RemoveContainer" containerID="ff6a6b8e5c0526f3b7d8c36047115459316f4c9d55fc82ecdd5161caeebb596e" Jul 29 08:20:10.993177 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.993068 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f545848f-cjcpw" event={"ID":"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a","Type":"ContainerStarted","Data":"76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db"} Jul 29 08:20:10.993177 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.993112 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f545848f-cjcpw" event={"ID":"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a","Type":"ContainerStarted","Data":"1e93cc38738af23d2b5bf829cc824848972d072ea10d1b13d4c648f55c797511"} Jul 29 08:20:10.996020 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:10.995890 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5pfnw" event={"ID":"30908914-7c33-4a7b-832a-bbf63c72b09b","Type":"ContainerStarted","Data":"9eb57d8a746867a33719b35ceaa96fcf18348623c2b68caa4cc5be1435b78d6b"} Jul 29 08:20:11.007324 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:11.007270 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-xwpsg" podStartSLOduration=1.178146128 podStartE2EDuration="20.007251098s" totalImagesPullingTime="18.82910497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:20:11.004405463 +0000 UTC m=+131.317586323" watchObservedRunningTime="2026-07-29 08:20:11.007251098 +0000 UTC m=+131.320431961" Jul 29 08:20:11.045092 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:11.043271 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f545848f-cjcpw" podStartSLOduration=4.043250862 podStartE2EDuration="4.043250862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:20:11.04274327 +0000 UTC m=+131.355924141" watchObservedRunningTime="2026-07-29 08:20:11.043250862 +0000 UTC m=+131.356431724" Jul 29 08:20:11.778161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:11.778123 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:11.887295 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:11.886591 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:12.002607 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:12.002569 2582 generic.go:356] "Generic (PLEG): container finished" podID="4c589bb2-2f4a-448e-a03b-2d1579ce3f62" containerID="a2e70a81bd3c7c13084e1cf3054b8ca5fcefc5632eadcb0c54fac5334277d16b" exitCode=0 Jul 29 08:20:12.003082 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:12.002647 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" event={"ID":"4c589bb2-2f4a-448e-a03b-2d1579ce3f62","Type":"ContainerDied","Data":"a2e70a81bd3c7c13084e1cf3054b8ca5fcefc5632eadcb0c54fac5334277d16b"} Jul 29 08:20:12.003471 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:12.003449 2582 scope.go:117] "RemoveContainer" containerID="a2e70a81bd3c7c13084e1cf3054b8ca5fcefc5632eadcb0c54fac5334277d16b" Jul 29 08:20:12.005435 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:12.005400 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5pfnw" event={"ID":"30908914-7c33-4a7b-832a-bbf63c72b09b","Type":"ContainerStarted","Data":"2b96ebe5e7e596968228bbb0a0ea360c126ec36080966b645df618fd45fcb1d4"} Jul 29 08:20:12.005533 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:12.005468 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5pfnw" event={"ID":"30908914-7c33-4a7b-832a-bbf63c72b09b","Type":"ContainerStarted","Data":"93a84fc83b6c37c7f6a5d08ccb26295f5a68ae51725d448417dde799d98c02fc"} Jul 29 08:20:12.008766 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:12.008693 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jqgwk" event={"ID":"d99f40c8-58ca-4ff1-99ea-c18951b03e15","Type":"ContainerStarted","Data":"f8e3de299c8e423296c2dd076359eabc1686acbc897d74624c43f3a04ba228ab"} Jul 29 08:20:12.011935 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:12.011915 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-xwpsg" Jul 29 08:20:12.036455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:12.036393 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5pfnw" podStartSLOduration=129.863006104 podStartE2EDuration="2m11.036371036s" totalImagesPullingTime="1.173364932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:20:12.034725389 +0000 UTC m=+132.347906251" watchObservedRunningTime="2026-07-29 08:20:12.036371036 +0000 UTC m=+132.349551899" Jul 29 08:20:12.141894 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:12.141832 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:12.201166 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:12.201082 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:13.016403 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:13.016361 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh" event={"ID":"4c589bb2-2f4a-448e-a03b-2d1579ce3f62","Type":"ContainerStarted","Data":"fbf9e6bded250106bb5583a07c27cc87db7597b2dd02e18b14bce830522d3229"} Jul 29 08:20:17.212168 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:17.212130 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:17.311208 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:17.311169 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:17.791454 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:17.791419 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:17.791679 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:17.791487 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:17.797120 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:17.797097 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:18.045670 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:18.045554 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:20:18.098646 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:18.098583 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b78c9495-pbbhc"] Jul 29 08:20:22.837634 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:22.837582 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:20:22.841604 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:22.841573 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5848859c-49qsj" Jul 29 08:20:34.068855 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:34.068819 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:34.171870 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:34.171834 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:36.969720 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:36.969682 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:43.125481 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.125421 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b78c9495-pbbhc" podUID="37ceb931-70a0-44ae-86f1-b978d171bd31" containerName="console" containerID="cri-o://11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9" gracePeriod=15 Jul 29 08:20:43.412708 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.412672 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:20:43.494291 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.494251 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-oauth-config\") pod \"37ceb931-70a0-44ae-86f1-b978d171bd31\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " Jul 29 08:20:43.494491 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.494310 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-console-config\") pod \"37ceb931-70a0-44ae-86f1-b978d171bd31\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " Jul 29 08:20:43.494491 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.494364 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-oauth-serving-cert\") pod \"37ceb931-70a0-44ae-86f1-b978d171bd31\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " Jul 29 08:20:43.494491 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.494399 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bctjw\" (UniqueName: \"kubernetes.io/projected/37ceb931-70a0-44ae-86f1-b978d171bd31-kube-api-access-bctjw\") pod \"37ceb931-70a0-44ae-86f1-b978d171bd31\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " Jul 29 08:20:43.494491 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.494421 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-serving-cert\") pod \"37ceb931-70a0-44ae-86f1-b978d171bd31\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " Jul 29 08:20:43.494491 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.494448 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-service-ca\") pod \"37ceb931-70a0-44ae-86f1-b978d171bd31\" (UID: \"37ceb931-70a0-44ae-86f1-b978d171bd31\") " Jul 29 08:20:43.494815 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.494780 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-console-config" (OuterVolumeSpecName: "console-config") pod "37ceb931-70a0-44ae-86f1-b978d171bd31" (UID: "37ceb931-70a0-44ae-86f1-b978d171bd31"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:20:43.494952 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.494812 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "37ceb931-70a0-44ae-86f1-b978d171bd31" (UID: "37ceb931-70a0-44ae-86f1-b978d171bd31"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:20:43.495051 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.495028 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-service-ca" (OuterVolumeSpecName: "service-ca") pod "37ceb931-70a0-44ae-86f1-b978d171bd31" (UID: "37ceb931-70a0-44ae-86f1-b978d171bd31"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:20:43.496777 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.496747 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "37ceb931-70a0-44ae-86f1-b978d171bd31" (UID: "37ceb931-70a0-44ae-86f1-b978d171bd31"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:20:43.496878 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.496824 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "37ceb931-70a0-44ae-86f1-b978d171bd31" (UID: "37ceb931-70a0-44ae-86f1-b978d171bd31"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:20:43.496934 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.496889 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ceb931-70a0-44ae-86f1-b978d171bd31-kube-api-access-bctjw" (OuterVolumeSpecName: "kube-api-access-bctjw") pod "37ceb931-70a0-44ae-86f1-b978d171bd31" (UID: "37ceb931-70a0-44ae-86f1-b978d171bd31"). InnerVolumeSpecName "kube-api-access-bctjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:20:43.595798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.595754 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-oauth-serving-cert\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:43.595798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.595790 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bctjw\" (UniqueName: \"kubernetes.io/projected/37ceb931-70a0-44ae-86f1-b978d171bd31-kube-api-access-bctjw\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:43.595798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.595804 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-serving-cert\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:43.596033 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.595818 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-service-ca\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:43.596033 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.595829 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/37ceb931-70a0-44ae-86f1-b978d171bd31-console-oauth-config\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:43.596033 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:43.595841 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/37ceb931-70a0-44ae-86f1-b978d171bd31-console-config\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:44.145430 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:44.145391 2582 generic.go:356] "Generic (PLEG): container finished" podID="37ceb931-70a0-44ae-86f1-b978d171bd31" containerID="11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9" exitCode=0 Jul 29 08:20:44.145935 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:44.145472 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b78c9495-pbbhc" event={"ID":"37ceb931-70a0-44ae-86f1-b978d171bd31","Type":"ContainerDied","Data":"11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9"} Jul 29 08:20:44.145935 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:44.145579 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b78c9495-pbbhc" event={"ID":"37ceb931-70a0-44ae-86f1-b978d171bd31","Type":"ContainerDied","Data":"ab2c65f0eedd8ab26bb0aaaa74b9c7d1f05074fa9577eec7a304a22b9be0eef9"} Jul 29 08:20:44.145935 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:44.145583 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b78c9495-pbbhc" Jul 29 08:20:44.145935 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:44.145659 2582 scope.go:117] "RemoveContainer" containerID="11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9" Jul 29 08:20:44.156138 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:44.156096 2582 scope.go:117] "RemoveContainer" containerID="11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9" Jul 29 08:20:44.156494 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:20:44.156472 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9\": container with ID starting with 11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9 not found: ID does not exist" containerID="11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9" Jul 29 08:20:44.156580 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:44.156501 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9"} err="failed to get container status \"11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9\": rpc error: code = NotFound desc = could not find container \"11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9\": container with ID starting with 11aff3efcbe45f12731c62aa7b43a6fab676079ee228e2a2da8e234aa1838ef9 not found: ID does not exist" Jul 29 08:20:44.168592 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:44.168561 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b78c9495-pbbhc"] Jul 29 08:20:44.173008 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:44.172986 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b78c9495-pbbhc"] Jul 29 08:20:44.251406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:44.251374 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ceb931-70a0-44ae-86f1-b978d171bd31" path="/var/lib/kubelet/pods/37ceb931-70a0-44ae-86f1-b978d171bd31/volumes" Jul 29 08:20:57.276392 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:57.276358 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:57.620217 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:57.620123 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:20:58.325857 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:58.325821 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:20:58.326262 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:58.326122 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="alertmanager" containerID="cri-o://641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7" gracePeriod=120 Jul 29 08:20:58.326262 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:58.326190 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy-web" containerID="cri-o://2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e" gracePeriod=120 Jul 29 08:20:58.326262 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:58.326215 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="prom-label-proxy" containerID="cri-o://8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece" gracePeriod=120 Jul 29 08:20:58.326262 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:58.326221 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy" containerID="cri-o://1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074" gracePeriod=120 Jul 29 08:20:58.326470 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:58.326190 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy-metric" containerID="cri-o://df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58" gracePeriod=120 Jul 29 08:20:58.326470 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:58.326262 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="config-reloader" containerID="cri-o://a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8" gracePeriod=120 Jul 29 08:20:59.212469 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.212426 2582 generic.go:356] "Generic (PLEG): container finished" podID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerID="8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece" exitCode=0 Jul 29 08:20:59.212469 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.212459 2582 generic.go:356] "Generic (PLEG): container finished" podID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerID="1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074" exitCode=0 Jul 29 08:20:59.212469 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.212467 2582 generic.go:356] "Generic (PLEG): container finished" podID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerID="a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8" exitCode=0 Jul 29 08:20:59.212469 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.212475 2582 generic.go:356] "Generic (PLEG): container finished" podID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerID="641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7" exitCode=0 Jul 29 08:20:59.212854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.212492 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerDied","Data":"8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece"} Jul 29 08:20:59.212854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.212559 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerDied","Data":"1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074"} Jul 29 08:20:59.212854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.212586 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerDied","Data":"a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8"} Jul 29 08:20:59.212854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.212610 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerDied","Data":"641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7"} Jul 29 08:20:59.574108 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.574081 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:20:59.742383 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742343 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.742383 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742389 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-cluster-tls-config\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.742688 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742418 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-out\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.742688 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742441 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-volume\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.742688 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742584 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-web\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.742852 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742689 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-metrics-client-ca\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.742852 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742741 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-web-config\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.742852 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742769 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rd6h\" (UniqueName: \"kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-kube-api-access-7rd6h\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.742852 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742805 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-main-db\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.742852 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742842 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-main-tls\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.743221 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742876 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-tls-assets\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.743221 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742924 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-trusted-ca-bundle\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.743221 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.742952 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\" (UID: \"4f14ba66-4b09-4afa-8993-8d4ed54f40d2\") " Jul 29 08:20:59.743221 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.743097 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:20:59.743444 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.743321 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-metrics-client-ca\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.743839 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.743812 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:20:59.745406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.745378 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:20:59.746057 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.745832 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-volume" (OuterVolumeSpecName: "config-volume") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:20:59.746151 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.746119 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:20:59.746332 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.746253 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:20:59.746332 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.746299 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-out" (OuterVolumeSpecName: "config-out") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:20:59.746332 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.746325 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:20:59.746517 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.746363 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:20:59.748153 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.748118 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:20:59.748232 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.748157 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-kube-api-access-7rd6h" (OuterVolumeSpecName: "kube-api-access-7rd6h") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "kube-api-access-7rd6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:20:59.749797 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.749775 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:20:59.757044 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.757022 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-web-config" (OuterVolumeSpecName: "web-config") pod "4f14ba66-4b09-4afa-8993-8d4ed54f40d2" (UID: "4f14ba66-4b09-4afa-8993-8d4ed54f40d2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:20:59.844323 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844286 2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-web-config\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844323 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844318 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rd6h\" (UniqueName: \"kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-kube-api-access-7rd6h\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844323 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844330 2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-main-db\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844341 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-main-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844352 2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-tls-assets\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844365 2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844378 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844391 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844403 2582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-cluster-tls-config\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844413 2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-out\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844424 2582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-config-volume\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:20:59.844566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:20:59.844434 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f14ba66-4b09-4afa-8993-8d4ed54f40d2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:21:00.219341 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.219308 2582 generic.go:356] "Generic (PLEG): container finished" podID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerID="df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58" exitCode=0 Jul 29 08:21:00.219341 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.219334 2582 generic.go:356] "Generic (PLEG): container finished" podID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerID="2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e" exitCode=0 Jul 29 08:21:00.219539 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.219374 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerDied","Data":"df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58"} Jul 29 08:21:00.219539 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.219397 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.219539 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.219473 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerDied","Data":"2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e"} Jul 29 08:21:00.219539 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.219505 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4f14ba66-4b09-4afa-8993-8d4ed54f40d2","Type":"ContainerDied","Data":"c88ebea03fa028e4a63f9cb2637ace129ca79c55ed47c3247da795da073647e8"} Jul 29 08:21:00.219696 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.219542 2582 scope.go:117] "RemoveContainer" containerID="8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece" Jul 29 08:21:00.228571 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.228555 2582 scope.go:117] "RemoveContainer" containerID="df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58" Jul 29 08:21:00.236677 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.236651 2582 scope.go:117] "RemoveContainer" containerID="1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074" Jul 29 08:21:00.244000 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.243976 2582 scope.go:117] "RemoveContainer" containerID="2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e" Jul 29 08:21:00.245541 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.245522 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:21:00.252681 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.252641 2582 scope.go:117] "RemoveContainer" containerID="a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8" Jul 29 08:21:00.254092 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.254071 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:21:00.260397 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.260380 2582 scope.go:117] "RemoveContainer" containerID="641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7" Jul 29 08:21:00.268066 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.268049 2582 scope.go:117] "RemoveContainer" containerID="a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89" Jul 29 08:21:00.275467 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.275446 2582 scope.go:117] "RemoveContainer" containerID="8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece" Jul 29 08:21:00.275795 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:21:00.275772 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece\": container with ID starting with 8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece not found: ID does not exist" containerID="8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece" Jul 29 08:21:00.275894 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.275801 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece"} err="failed to get container status \"8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece\": rpc error: code = NotFound desc = could not find container \"8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece\": container with ID starting with 8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece not found: ID does not exist" Jul 29 08:21:00.275894 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.275824 2582 scope.go:117] "RemoveContainer" containerID="df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58" Jul 29 08:21:00.276062 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:21:00.276046 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58\": container with ID starting with df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58 not found: ID does not exist" containerID="df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58" Jul 29 08:21:00.276100 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.276066 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58"} err="failed to get container status \"df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58\": rpc error: code = NotFound desc = could not find container \"df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58\": container with ID starting with df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58 not found: ID does not exist" Jul 29 08:21:00.276100 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.276080 2582 scope.go:117] "RemoveContainer" containerID="1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074" Jul 29 08:21:00.276342 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:21:00.276321 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074\": container with ID starting with 1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074 not found: ID does not exist" containerID="1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074" Jul 29 08:21:00.276397 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.276349 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074"} err="failed to get container status \"1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074\": rpc error: code = NotFound desc = could not find container \"1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074\": container with ID starting with 1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074 not found: ID does not exist" Jul 29 08:21:00.276397 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.276371 2582 scope.go:117] "RemoveContainer" containerID="2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e" Jul 29 08:21:00.276599 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:21:00.276581 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e\": container with ID starting with 2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e not found: ID does not exist" containerID="2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e" Jul 29 08:21:00.276719 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.276599 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e"} err="failed to get container status \"2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e\": rpc error: code = NotFound desc = could not find container \"2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e\": container with ID starting with 2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e not found: ID does not exist" Jul 29 08:21:00.276719 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.276611 2582 scope.go:117] "RemoveContainer" containerID="a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8" Jul 29 08:21:00.276888 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:21:00.276871 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8\": container with ID starting with a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8 not found: ID does not exist" containerID="a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8" Jul 29 08:21:00.276928 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.276893 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8"} err="failed to get container status \"a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8\": rpc error: code = NotFound desc = could not find container \"a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8\": container with ID starting with a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8 not found: ID does not exist" Jul 29 08:21:00.276928 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.276908 2582 scope.go:117] "RemoveContainer" containerID="641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7" Jul 29 08:21:00.277143 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:21:00.277127 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7\": container with ID starting with 641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7 not found: ID does not exist" containerID="641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7" Jul 29 08:21:00.277193 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.277147 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7"} err="failed to get container status \"641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7\": rpc error: code = NotFound desc = could not find container \"641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7\": container with ID starting with 641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7 not found: ID does not exist" Jul 29 08:21:00.277193 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.277159 2582 scope.go:117] "RemoveContainer" containerID="a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89" Jul 29 08:21:00.277378 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:21:00.277360 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89\": container with ID starting with a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89 not found: ID does not exist" containerID="a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89" Jul 29 08:21:00.277416 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.277381 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89"} err="failed to get container status \"a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89\": rpc error: code = NotFound desc = could not find container \"a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89\": container with ID starting with a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89 not found: ID does not exist" Jul 29 08:21:00.277416 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.277395 2582 scope.go:117] "RemoveContainer" containerID="8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece" Jul 29 08:21:00.277596 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.277579 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece"} err="failed to get container status \"8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece\": rpc error: code = NotFound desc = could not find container \"8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece\": container with ID starting with 8cc46c55501fff31fa00f223234308d0f23e2d82cbc721000504ec9ed7082ece not found: ID does not exist" Jul 29 08:21:00.277657 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.277597 2582 scope.go:117] "RemoveContainer" containerID="df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58" Jul 29 08:21:00.277816 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.277800 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58"} err="failed to get container status \"df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58\": rpc error: code = NotFound desc = could not find container \"df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58\": container with ID starting with df1f9519ac500c0a84e02cacbb0724acf65d07fc689741fa8bb0d43f105dea58 not found: ID does not exist" Jul 29 08:21:00.277856 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.277816 2582 scope.go:117] "RemoveContainer" containerID="1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074" Jul 29 08:21:00.277994 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.277978 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074"} err="failed to get container status \"1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074\": rpc error: code = NotFound desc = could not find container \"1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074\": container with ID starting with 1b406cd79612566ab3271ad15dc06f840ca9431f771abb63a30897661fd72074 not found: ID does not exist" Jul 29 08:21:00.278030 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.277995 2582 scope.go:117] "RemoveContainer" containerID="2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e" Jul 29 08:21:00.278174 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.278160 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e"} err="failed to get container status \"2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e\": rpc error: code = NotFound desc = could not find container \"2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e\": container with ID starting with 2c467f47d5714795dab5415670dd1c9a5848c39a64fb7688b67241b71f444b4e not found: ID does not exist" Jul 29 08:21:00.278221 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.278174 2582 scope.go:117] "RemoveContainer" containerID="a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8" Jul 29 08:21:00.278351 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.278337 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8"} err="failed to get container status \"a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8\": rpc error: code = NotFound desc = could not find container \"a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8\": container with ID starting with a441f3fd956bbe70a1a078632326816c0c7e8b83b54ab86d422271a0d4106ad8 not found: ID does not exist" Jul 29 08:21:00.278351 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.278350 2582 scope.go:117] "RemoveContainer" containerID="641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7" Jul 29 08:21:00.278515 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.278498 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7"} err="failed to get container status \"641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7\": rpc error: code = NotFound desc = could not find container \"641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7\": container with ID starting with 641f6104808584c5c3c4b92d4a7bd52db09815bd9c4c20e4b2d9fbb2174871e7 not found: ID does not exist" Jul 29 08:21:00.278554 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.278516 2582 scope.go:117] "RemoveContainer" containerID="a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89" Jul 29 08:21:00.278713 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.278695 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89"} err="failed to get container status \"a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89\": rpc error: code = NotFound desc = could not find container \"a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89\": container with ID starting with a3c26b414c99e0d97b2de026df0c77184bb080df37dae9e0dc487448f62c7b89 not found: ID does not exist" Jul 29 08:21:00.285464 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285443 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:21:00.285738 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285726 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="config-reloader" Jul 29 08:21:00.285784 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285741 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="config-reloader" Jul 29 08:21:00.285784 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285749 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="prom-label-proxy" Jul 29 08:21:00.285784 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285755 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="prom-label-proxy" Jul 29 08:21:00.285784 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285764 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy" Jul 29 08:21:00.285784 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285770 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy" Jul 29 08:21:00.285784 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285777 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy-web" Jul 29 08:21:00.285784 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285784 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy-web" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285794 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="init-config-reloader" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285800 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="init-config-reloader" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285810 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="alertmanager" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285815 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="alertmanager" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285821 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ceb931-70a0-44ae-86f1-b978d171bd31" containerName="console" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285826 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ceb931-70a0-44ae-86f1-b978d171bd31" containerName="console" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285831 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy-metric" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285836 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy-metric" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285883 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy-web" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285890 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="alertmanager" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285896 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="config-reloader" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285903 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ceb931-70a0-44ae-86f1-b978d171bd31" containerName="console" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285909 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285915 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="kube-rbac-proxy-metric" Jul 29 08:21:00.285978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.285922 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" containerName="prom-label-proxy" Jul 29 08:21:00.290715 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.290699 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.293988 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.293971 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 08:21:00.294218 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.294206 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 08:21:00.294508 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.294489 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 08:21:00.294605 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.294489 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 08:21:00.294605 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.294512 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 08:21:00.294818 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.294804 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 08:21:00.294898 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.294817 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 08:21:00.295374 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.295359 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 08:21:00.296340 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.296320 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v4wpr\"" Jul 29 08:21:00.302087 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.302069 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 08:21:00.308327 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.308304 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:21:00.448708 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448669 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c554ac19-c275-480a-9b1f-c526717e33c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.448708 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448710 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.448906 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448733 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.448906 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-web-config\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.448906 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448795 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.448906 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448835 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.448906 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c554ac19-c275-480a-9b1f-c526717e33c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.448906 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448886 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c554ac19-c275-480a-9b1f-c526717e33c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.449088 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.449088 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448955 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.449088 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.448976 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c554ac19-c275-480a-9b1f-c526717e33c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.449088 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.449005 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c554ac19-c275-480a-9b1f-c526717e33c5-config-out\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.449088 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.449044 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfb6\" (UniqueName: \"kubernetes.io/projected/c554ac19-c275-480a-9b1f-c526717e33c5-kube-api-access-csfb6\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.549401 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.549371 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.549558 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.549410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c554ac19-c275-480a-9b1f-c526717e33c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.549558 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.549443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c554ac19-c275-480a-9b1f-c526717e33c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.549558 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.549463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.549558 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.549486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.549558 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.549514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c554ac19-c275-480a-9b1f-c526717e33c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.549558 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.549537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c554ac19-c275-480a-9b1f-c526717e33c5-config-out\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.549916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.549566 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfb6\" (UniqueName: \"kubernetes.io/projected/c554ac19-c275-480a-9b1f-c526717e33c5-kube-api-access-csfb6\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.549991 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.549935 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c554ac19-c275-480a-9b1f-c526717e33c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.550100 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.550064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c554ac19-c275-480a-9b1f-c526717e33c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.550328 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.550113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.550328 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.550149 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.550427 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.550273 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-web-config\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.550427 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.550370 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.552807 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.552769 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.552807 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.552800 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.554099 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.553781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c554ac19-c275-480a-9b1f-c526717e33c5-config-out\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.554099 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.554028 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c554ac19-c275-480a-9b1f-c526717e33c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.554099 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.550317 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c554ac19-c275-480a-9b1f-c526717e33c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.554099 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.554059 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.554716 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.554530 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c554ac19-c275-480a-9b1f-c526717e33c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.554716 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.554643 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.554716 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.554677 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.555802 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.555784 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-web-config\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.556851 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.556833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c554ac19-c275-480a-9b1f-c526717e33c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.559723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.559695 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfb6\" (UniqueName: \"kubernetes.io/projected/c554ac19-c275-480a-9b1f-c526717e33c5-kube-api-access-csfb6\") pod \"alertmanager-main-0\" (UID: \"c554ac19-c275-480a-9b1f-c526717e33c5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.599816 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.599778 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:21:00.743857 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:00.743833 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:21:00.746193 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:21:00.746169 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc554ac19_c275_480a_9b1f_c526717e33c5.slice/crio-334c1848146648e7d9b44246114cd273e0793dc2587bca43be20c2e793dd48dd WatchSource:0}: Error finding container 334c1848146648e7d9b44246114cd273e0793dc2587bca43be20c2e793dd48dd: Status 404 returned error can't find the container with id 334c1848146648e7d9b44246114cd273e0793dc2587bca43be20c2e793dd48dd Jul 29 08:21:01.225346 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:01.225313 2582 generic.go:356] "Generic (PLEG): container finished" podID="c554ac19-c275-480a-9b1f-c526717e33c5" containerID="163de5511c5985d4f275a5c87d0a7a3e36a2c33596bab8d58770c0bc78865a27" exitCode=0 Jul 29 08:21:01.225561 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:01.225355 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c554ac19-c275-480a-9b1f-c526717e33c5","Type":"ContainerDied","Data":"163de5511c5985d4f275a5c87d0a7a3e36a2c33596bab8d58770c0bc78865a27"} Jul 29 08:21:01.225561 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:01.225419 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c554ac19-c275-480a-9b1f-c526717e33c5","Type":"ContainerStarted","Data":"334c1848146648e7d9b44246114cd273e0793dc2587bca43be20c2e793dd48dd"} Jul 29 08:21:02.232524 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.232486 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c554ac19-c275-480a-9b1f-c526717e33c5","Type":"ContainerStarted","Data":"e582362e336822ca27ef6c3077f0af4d982e38c2e31b5311d040c446a8c9673c"} Jul 29 08:21:02.233040 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.232564 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c554ac19-c275-480a-9b1f-c526717e33c5","Type":"ContainerStarted","Data":"9791519b614af216d6d21c355ccb6bf90874d7d034d8d1baee08d90efbc3af89"} Jul 29 08:21:02.233040 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.232610 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c554ac19-c275-480a-9b1f-c526717e33c5","Type":"ContainerStarted","Data":"d43790ebada04de405a7d50fefbd1b988395a078fdd043089456b68196d6cf1a"} Jul 29 08:21:02.233040 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.232676 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c554ac19-c275-480a-9b1f-c526717e33c5","Type":"ContainerStarted","Data":"8ea7e3b526314cc38a07a05a3144e01f721e93962ec02a9f7149eb4e03e6df55"} Jul 29 08:21:02.233040 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.232724 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c554ac19-c275-480a-9b1f-c526717e33c5","Type":"ContainerStarted","Data":"6ac423d4bbf9098ea2f8cadd3af7f1eacf13f8f03cb69931d2db2f870d0fa1a1"} Jul 29 08:21:02.233040 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.232778 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c554ac19-c275-480a-9b1f-c526717e33c5","Type":"ContainerStarted","Data":"f924ea3a0a2b44c2cdc5bd606cdcd7241c35363ed5022e15d2e8466c15bde3d3"} Jul 29 08:21:02.251007 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.250977 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f14ba66-4b09-4afa-8993-8d4ed54f40d2" path="/var/lib/kubelet/pods/4f14ba66-4b09-4afa-8993-8d4ed54f40d2/volumes" Jul 29 08:21:02.263594 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.263535 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.247943506 podStartE2EDuration="2.263514545s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.571039ms" isStatefulPod=false podCreationTimestamp="2026-07-29 08:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:02.261116966 +0000 UTC m=+182.574297866" watchObservedRunningTime="2026-07-29 08:21:02.263514545 +0000 UTC m=+182.576695406" Jul 29 08:21:02.393730 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.393701 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6b44fd54c6-n55ng"] Jul 29 08:21:02.396270 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.396255 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.398730 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.398695 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5b5s2\"" Jul 29 08:21:02.398730 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.398707 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 29 08:21:02.398730 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.398703 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 29 08:21:02.399033 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.398782 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 29 08:21:02.399033 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.398862 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 29 08:21:02.399154 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.399096 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 29 08:21:02.404820 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.404798 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 29 08:21:02.441047 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.438542 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b44fd54c6-n55ng"] Jul 29 08:21:02.567489 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.567368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ad9287-997b-4533-8ac7-7b063a4a1664-metrics-client-ca\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.567489 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.567466 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckp95\" (UniqueName: \"kubernetes.io/projected/95ad9287-997b-4533-8ac7-7b063a4a1664-kube-api-access-ckp95\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.567749 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.567501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ad9287-997b-4533-8ac7-7b063a4a1664-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.567749 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.567531 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-telemeter-client-tls\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.567749 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.567595 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ad9287-997b-4533-8ac7-7b063a4a1664-serving-certs-ca-bundle\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.567749 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.567691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-federate-client-tls\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.567749 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.567742 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.567927 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.567786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-secret-telemeter-client\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.669021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.668981 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ad9287-997b-4533-8ac7-7b063a4a1664-metrics-client-ca\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.669183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.669041 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckp95\" (UniqueName: \"kubernetes.io/projected/95ad9287-997b-4533-8ac7-7b063a4a1664-kube-api-access-ckp95\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.669183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.669060 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ad9287-997b-4533-8ac7-7b063a4a1664-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.669183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.669083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-telemeter-client-tls\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.669183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.669102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ad9287-997b-4533-8ac7-7b063a4a1664-serving-certs-ca-bundle\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.669183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.669117 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-federate-client-tls\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.669183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.669139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.669486 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.669187 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-secret-telemeter-client\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.669943 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.669905 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ad9287-997b-4533-8ac7-7b063a4a1664-metrics-client-ca\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.669943 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.669905 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ad9287-997b-4533-8ac7-7b063a4a1664-serving-certs-ca-bundle\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.670302 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.670235 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ad9287-997b-4533-8ac7-7b063a4a1664-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.672019 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.671996 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-telemeter-client-tls\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.672090 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.672046 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.672413 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.672392 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-secret-telemeter-client\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.672456 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.672392 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/95ad9287-997b-4533-8ac7-7b063a4a1664-federate-client-tls\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.677346 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.677328 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckp95\" (UniqueName: \"kubernetes.io/projected/95ad9287-997b-4533-8ac7-7b063a4a1664-kube-api-access-ckp95\") pod \"telemeter-client-6b44fd54c6-n55ng\" (UID: \"95ad9287-997b-4533-8ac7-7b063a4a1664\") " pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.706059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.706034 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" Jul 29 08:21:02.846566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:02.846540 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b44fd54c6-n55ng"] Jul 29 08:21:02.850198 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:21:02.850164 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ad9287_997b_4533_8ac7_7b063a4a1664.slice/crio-427a0a5a986c25dd9c15651f8262471f7dedebc61c6c2993699e10856efa0aee WatchSource:0}: Error finding container 427a0a5a986c25dd9c15651f8262471f7dedebc61c6c2993699e10856efa0aee: Status 404 returned error can't find the container with id 427a0a5a986c25dd9c15651f8262471f7dedebc61c6c2993699e10856efa0aee Jul 29 08:21:03.237439 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:03.237405 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" event={"ID":"95ad9287-997b-4533-8ac7-7b063a4a1664","Type":"ContainerStarted","Data":"427a0a5a986c25dd9c15651f8262471f7dedebc61c6c2993699e10856efa0aee"} Jul 29 08:21:05.248407 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:05.248374 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" event={"ID":"95ad9287-997b-4533-8ac7-7b063a4a1664","Type":"ContainerStarted","Data":"7569684fd43a21366745db13be5a569832a910a6fa1a13acdde7a35b73f1132c"} Jul 29 08:21:05.248786 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:05.248425 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" event={"ID":"95ad9287-997b-4533-8ac7-7b063a4a1664","Type":"ContainerStarted","Data":"ecc8ec9c590b9f0b4e2bdd354a12bd5f89b441e1612a8577a28a9911fc10901a"} Jul 29 08:21:05.248786 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:05.248448 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" event={"ID":"95ad9287-997b-4533-8ac7-7b063a4a1664","Type":"ContainerStarted","Data":"c0049ff14913cf20271e119959f46ba9db0fe76cd0726b18ec92eb7a04c7f820"} Jul 29 08:21:05.270203 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:05.270148 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6b44fd54c6-n55ng" podStartSLOduration=1.450655103 podStartE2EDuration="3.270132574s" totalImagesPullingTime="1.819477471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:05.269247355 +0000 UTC m=+185.582428205" watchObservedRunningTime="2026-07-29 08:21:05.270132574 +0000 UTC m=+185.583313433" Jul 29 08:21:06.126710 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.126680 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77465cf9d5-q2jqq"] Jul 29 08:21:06.129028 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.129009 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.145070 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.145044 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77465cf9d5-q2jqq"] Jul 29 08:21:06.204489 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.204454 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:21:06.301290 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.301243 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76dl\" (UniqueName: \"kubernetes.io/projected/af0f307e-22fe-4ca2-a69f-c31f3bdee846-kube-api-access-p76dl\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.301723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.301308 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-config\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.301723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.301335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-oauth-serving-cert\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.301723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.301399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-trusted-ca-bundle\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.301723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.301422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-oauth-config\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.301723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.301439 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-service-ca\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.301723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.301496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-serving-cert\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.303438 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.303413 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:21:06.402872 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.402783 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76dl\" (UniqueName: \"kubernetes.io/projected/af0f307e-22fe-4ca2-a69f-c31f3bdee846-kube-api-access-p76dl\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.402872 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.402826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-config\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.402872 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.402860 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-oauth-serving-cert\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.403241 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.403214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-trusted-ca-bundle\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.403313 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.403282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-oauth-config\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.403382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.403325 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-service-ca\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.403468 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.403428 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-serving-cert\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.403658 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.403602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-config\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.404193 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.404166 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-service-ca\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.404290 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.404206 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-oauth-serving-cert\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.404290 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.404256 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-trusted-ca-bundle\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.406025 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.405999 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-serving-cert\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.406106 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.406023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-oauth-config\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.413553 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.413533 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76dl\" (UniqueName: \"kubernetes.io/projected/af0f307e-22fe-4ca2-a69f-c31f3bdee846-kube-api-access-p76dl\") pod \"console-77465cf9d5-q2jqq\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.438171 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.438136 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:06.572422 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:06.572399 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77465cf9d5-q2jqq"] Jul 29 08:21:06.574953 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:21:06.574925 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0f307e_22fe_4ca2_a69f_c31f3bdee846.slice/crio-3f129253edbc468afdea27997fd673e7eee1ec8113d4c3a462d38f3c37673235 WatchSource:0}: Error finding container 3f129253edbc468afdea27997fd673e7eee1ec8113d4c3a462d38f3c37673235: Status 404 returned error can't find the container with id 3f129253edbc468afdea27997fd673e7eee1ec8113d4c3a462d38f3c37673235 Jul 29 08:21:07.262235 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:07.262193 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77465cf9d5-q2jqq" event={"ID":"af0f307e-22fe-4ca2-a69f-c31f3bdee846","Type":"ContainerStarted","Data":"b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15"} Jul 29 08:21:07.262421 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:07.262249 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77465cf9d5-q2jqq" event={"ID":"af0f307e-22fe-4ca2-a69f-c31f3bdee846","Type":"ContainerStarted","Data":"3f129253edbc468afdea27997fd673e7eee1ec8113d4c3a462d38f3c37673235"} Jul 29 08:21:07.281849 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:07.281798 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77465cf9d5-q2jqq" podStartSLOduration=1.281781807 podStartE2EDuration="1.281781807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:07.280387377 +0000 UTC m=+187.593568238" watchObservedRunningTime="2026-07-29 08:21:07.281781807 +0000 UTC m=+187.594962667" Jul 29 08:21:16.438313 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:16.438272 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:16.438834 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:16.438354 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:16.443414 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:16.443390 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:17.304377 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:17.304351 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:21:17.371125 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:17.371089 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f545848f-cjcpw"] Jul 29 08:21:17.493827 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:17.493796 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:21:17.593344 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:17.593249 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:21:17.658978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:17.658935 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:21:42.392206 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.392144 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f545848f-cjcpw" podUID="2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" containerName="console" containerID="cri-o://76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db" gracePeriod=15 Jul 29 08:21:42.639511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.639485 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:21:42.715799 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.715770 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-serving-cert\") pod \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " Jul 29 08:21:42.715957 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.715814 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sqrg\" (UniqueName: \"kubernetes.io/projected/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-kube-api-access-8sqrg\") pod \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " Jul 29 08:21:42.715957 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.715862 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-service-ca\") pod \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " Jul 29 08:21:42.715957 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.715883 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-trusted-ca-bundle\") pod \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " Jul 29 08:21:42.715957 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.715904 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-oauth-config\") pod \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " Jul 29 08:21:42.715957 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.715933 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-oauth-serving-cert\") pod \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " Jul 29 08:21:42.716197 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.715963 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-config\") pod \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\" (UID: \"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a\") " Jul 29 08:21:42.716444 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.716393 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" (UID: "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:21:42.716540 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.716471 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-config" (OuterVolumeSpecName: "console-config") pod "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" (UID: "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:21:42.716540 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.716477 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" (UID: "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:21:42.716849 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.716721 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" (UID: "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:21:42.718086 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.718062 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" (UID: "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:21:42.718184 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.718166 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-kube-api-access-8sqrg" (OuterVolumeSpecName: "kube-api-access-8sqrg") pod "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" (UID: "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a"). InnerVolumeSpecName "kube-api-access-8sqrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:21:42.718222 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.718208 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" (UID: "2b28c8fe-71fb-4242-8caa-cff2ee11dd3a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:21:42.816936 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.816898 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-oauth-serving-cert\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:21:42.816936 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.816931 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-config\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:21:42.816936 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.816940 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-serving-cert\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:21:42.817156 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.816949 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sqrg\" (UniqueName: \"kubernetes.io/projected/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-kube-api-access-8sqrg\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:21:42.817156 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.816958 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-service-ca\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:21:42.817156 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.816966 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-trusted-ca-bundle\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:21:42.817156 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:42.816975 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a-console-oauth-config\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:21:43.416885 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:43.416856 2582 generic.go:356] "Generic (PLEG): container finished" podID="2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" containerID="76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db" exitCode=0 Jul 29 08:21:43.417294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:43.416900 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f545848f-cjcpw" Jul 29 08:21:43.417294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:43.416911 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f545848f-cjcpw" event={"ID":"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a","Type":"ContainerDied","Data":"76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db"} Jul 29 08:21:43.417294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:43.416950 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f545848f-cjcpw" event={"ID":"2b28c8fe-71fb-4242-8caa-cff2ee11dd3a","Type":"ContainerDied","Data":"1e93cc38738af23d2b5bf829cc824848972d072ea10d1b13d4c648f55c797511"} Jul 29 08:21:43.417294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:43.416969 2582 scope.go:117] "RemoveContainer" containerID="76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db" Jul 29 08:21:43.426031 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:43.426015 2582 scope.go:117] "RemoveContainer" containerID="76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db" Jul 29 08:21:43.426286 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:21:43.426266 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db\": container with ID starting with 76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db not found: ID does not exist" containerID="76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db" Jul 29 08:21:43.426353 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:43.426300 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db"} err="failed to get container status \"76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db\": rpc error: code = NotFound desc = could not find container \"76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db\": container with ID starting with 76226dfd49cc74bdbe92cf7ed8f6ab504979f7419a99bf5b777d5650202091db not found: ID does not exist" Jul 29 08:21:43.439167 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:43.439136 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f545848f-cjcpw"] Jul 29 08:21:43.445467 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:43.445446 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f545848f-cjcpw"] Jul 29 08:21:44.251018 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:44.250987 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" path="/var/lib/kubelet/pods/2b28c8fe-71fb-4242-8caa-cff2ee11dd3a/volumes" Jul 29 08:21:59.720950 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:21:59.720910 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:12.626606 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:12.626562 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:12.754498 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:12.754466 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:13.283178 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.283150 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6975ffc864-nq5t5"] Jul 29 08:22:13.283498 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.283485 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" containerName="console" Jul 29 08:22:13.283546 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.283500 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" containerName="console" Jul 29 08:22:13.283579 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.283557 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b28c8fe-71fb-4242-8caa-cff2ee11dd3a" containerName="console" Jul 29 08:22:13.286581 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.286559 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.298729 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.298706 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6975ffc864-nq5t5"] Jul 29 08:22:13.351741 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.351707 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:13.386937 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.386911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-trusted-ca-bundle\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.387084 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.386954 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-serving-cert\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.387084 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.386971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-service-ca\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.387084 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.387052 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-oauth-config\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.387084 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.387078 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-oauth-serving-cert\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.387298 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.387111 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxkml\" (UniqueName: \"kubernetes.io/projected/8eda6475-9851-48cd-bbda-d62724b9fc1c-kube-api-access-rxkml\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.387298 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.387132 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-config\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.416868 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.416838 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:13.488531 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.488500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-trusted-ca-bundle\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.488722 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.488549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-serving-cert\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.488722 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.488568 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-service-ca\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.488808 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.488753 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-oauth-config\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.488808 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.488795 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-oauth-serving-cert\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.488871 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.488851 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxkml\" (UniqueName: \"kubernetes.io/projected/8eda6475-9851-48cd-bbda-d62724b9fc1c-kube-api-access-rxkml\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.488916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.488886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-config\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.489473 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.489450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-trusted-ca-bundle\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.489575 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.489486 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-service-ca\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.489575 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.489494 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-oauth-serving-cert\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.489836 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.489816 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-config\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.491114 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.491092 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-oauth-config\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.491338 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.491318 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-serving-cert\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.498505 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.498485 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxkml\" (UniqueName: \"kubernetes.io/projected/8eda6475-9851-48cd-bbda-d62724b9fc1c-kube-api-access-rxkml\") pod \"console-6975ffc864-nq5t5\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.595576 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.595477 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:13.733786 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:13.733761 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6975ffc864-nq5t5"] Jul 29 08:22:13.736073 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:22:13.736050 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eda6475_9851_48cd_bbda_d62724b9fc1c.slice/crio-ef30afccc759c02cd82edb78cc584e8e5f23a2068aedcb2388c377d5839ce3d5 WatchSource:0}: Error finding container ef30afccc759c02cd82edb78cc584e8e5f23a2068aedcb2388c377d5839ce3d5: Status 404 returned error can't find the container with id ef30afccc759c02cd82edb78cc584e8e5f23a2068aedcb2388c377d5839ce3d5 Jul 29 08:22:14.550103 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:14.550072 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6975ffc864-nq5t5" event={"ID":"8eda6475-9851-48cd-bbda-d62724b9fc1c","Type":"ContainerStarted","Data":"7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32"} Jul 29 08:22:14.550269 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:14.550116 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6975ffc864-nq5t5" event={"ID":"8eda6475-9851-48cd-bbda-d62724b9fc1c","Type":"ContainerStarted","Data":"ef30afccc759c02cd82edb78cc584e8e5f23a2068aedcb2388c377d5839ce3d5"} Jul 29 08:22:14.570418 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:14.570363 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6975ffc864-nq5t5" podStartSLOduration=1.570347633 podStartE2EDuration="1.570347633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:22:14.568227319 +0000 UTC m=+254.881408179" watchObservedRunningTime="2026-07-29 08:22:14.570347633 +0000 UTC m=+254.883528493" Jul 29 08:22:23.596028 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:23.595953 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:23.596028 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:23.596024 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:23.600590 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:23.600570 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:24.590823 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:24.590794 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:24.593330 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:24.593308 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:22:24.641637 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:24.641591 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77465cf9d5-q2jqq"] Jul 29 08:22:24.668066 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:24.668035 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:24.739959 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:24.739924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:24.844254 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:24.844174 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:24.946054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:24.946010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:34.082946 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:34.082908 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:34.190202 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:34.190165 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:36.878295 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:36.878249 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:36.962449 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:36.962411 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:22:49.663612 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:49.663544 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77465cf9d5-q2jqq" podUID="af0f307e-22fe-4ca2-a69f-c31f3bdee846" containerName="console" containerID="cri-o://b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15" gracePeriod=15 Jul 29 08:22:49.906245 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:49.906220 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:22:50.003924 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.003892 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p76dl\" (UniqueName: \"kubernetes.io/projected/af0f307e-22fe-4ca2-a69f-c31f3bdee846-kube-api-access-p76dl\") pod \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " Jul 29 08:22:50.004105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.003937 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-trusted-ca-bundle\") pod \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " Jul 29 08:22:50.004105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.003967 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-config\") pod \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " Jul 29 08:22:50.004105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.003989 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-oauth-serving-cert\") pod \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " Jul 29 08:22:50.004105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.004020 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-oauth-config\") pod \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " Jul 29 08:22:50.004105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.004068 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-service-ca\") pod \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " Jul 29 08:22:50.004105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.004105 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-serving-cert\") pod \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\" (UID: \"af0f307e-22fe-4ca2-a69f-c31f3bdee846\") " Jul 29 08:22:50.004511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.004454 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "af0f307e-22fe-4ca2-a69f-c31f3bdee846" (UID: "af0f307e-22fe-4ca2-a69f-c31f3bdee846"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:22:50.004511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.004467 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-config" (OuterVolumeSpecName: "console-config") pod "af0f307e-22fe-4ca2-a69f-c31f3bdee846" (UID: "af0f307e-22fe-4ca2-a69f-c31f3bdee846"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:22:50.004511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.004483 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "af0f307e-22fe-4ca2-a69f-c31f3bdee846" (UID: "af0f307e-22fe-4ca2-a69f-c31f3bdee846"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:22:50.004693 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.004513 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-service-ca" (OuterVolumeSpecName: "service-ca") pod "af0f307e-22fe-4ca2-a69f-c31f3bdee846" (UID: "af0f307e-22fe-4ca2-a69f-c31f3bdee846"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:22:50.006345 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.006322 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0f307e-22fe-4ca2-a69f-c31f3bdee846-kube-api-access-p76dl" (OuterVolumeSpecName: "kube-api-access-p76dl") pod "af0f307e-22fe-4ca2-a69f-c31f3bdee846" (UID: "af0f307e-22fe-4ca2-a69f-c31f3bdee846"). InnerVolumeSpecName "kube-api-access-p76dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:22:50.006434 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.006340 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "af0f307e-22fe-4ca2-a69f-c31f3bdee846" (UID: "af0f307e-22fe-4ca2-a69f-c31f3bdee846"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:22:50.006434 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.006369 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "af0f307e-22fe-4ca2-a69f-c31f3bdee846" (UID: "af0f307e-22fe-4ca2-a69f-c31f3bdee846"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:22:50.104723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.104692 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-config\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:22:50.104723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.104719 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-oauth-serving-cert\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:22:50.104723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.104728 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-oauth-config\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:22:50.104951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.104737 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-service-ca\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:22:50.104951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.104745 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af0f307e-22fe-4ca2-a69f-c31f3bdee846-console-serving-cert\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:22:50.104951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.104753 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p76dl\" (UniqueName: \"kubernetes.io/projected/af0f307e-22fe-4ca2-a69f-c31f3bdee846-kube-api-access-p76dl\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:22:50.104951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.104761 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0f307e-22fe-4ca2-a69f-c31f3bdee846-trusted-ca-bundle\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:22:50.694926 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.694850 2582 generic.go:356] "Generic (PLEG): container finished" podID="af0f307e-22fe-4ca2-a69f-c31f3bdee846" containerID="b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15" exitCode=0 Jul 29 08:22:50.694926 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.694892 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77465cf9d5-q2jqq" Jul 29 08:22:50.694926 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.694925 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77465cf9d5-q2jqq" event={"ID":"af0f307e-22fe-4ca2-a69f-c31f3bdee846","Type":"ContainerDied","Data":"b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15"} Jul 29 08:22:50.695407 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.694962 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77465cf9d5-q2jqq" event={"ID":"af0f307e-22fe-4ca2-a69f-c31f3bdee846","Type":"ContainerDied","Data":"3f129253edbc468afdea27997fd673e7eee1ec8113d4c3a462d38f3c37673235"} Jul 29 08:22:50.695407 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.694982 2582 scope.go:117] "RemoveContainer" containerID="b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15" Jul 29 08:22:50.703328 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.703306 2582 scope.go:117] "RemoveContainer" containerID="b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15" Jul 29 08:22:50.703593 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:22:50.703573 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15\": container with ID starting with b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15 not found: ID does not exist" containerID="b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15" Jul 29 08:22:50.703738 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.703604 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15"} err="failed to get container status \"b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15\": rpc error: code = NotFound desc = could not find container \"b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15\": container with ID starting with b56a37b72bdf79a80ca7521041687304c9c656ddb99e6a24766d6f979f86bc15 not found: ID does not exist" Jul 29 08:22:50.715890 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.715866 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77465cf9d5-q2jqq"] Jul 29 08:22:50.721082 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:50.721043 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77465cf9d5-q2jqq"] Jul 29 08:22:52.250425 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:22:52.250391 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af0f307e-22fe-4ca2-a69f-c31f3bdee846" path="/var/lib/kubelet/pods/af0f307e-22fe-4ca2-a69f-c31f3bdee846/volumes" Jul 29 08:23:00.095483 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:23:00.095457 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 08:23:47.729771 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:23:47.729723 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:04.919840 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:04.919793 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:05.016219 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.016184 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:05.629833 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.629799 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:05.643943 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.643912 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6"] Jul 29 08:24:05.644262 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.644250 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af0f307e-22fe-4ca2-a69f-c31f3bdee846" containerName="console" Jul 29 08:24:05.644301 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.644265 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af0f307e-22fe-4ca2-a69f-c31f3bdee846" containerName="console" Jul 29 08:24:05.644338 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.644329 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af0f307e-22fe-4ca2-a69f-c31f3bdee846" containerName="console" Jul 29 08:24:05.646569 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.646554 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" Jul 29 08:24:05.649069 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.649048 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 29 08:24:05.649735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.649720 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 29 08:24:05.649831 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.649761 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-j68k5\"" Jul 29 08:24:05.656260 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.656238 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6"] Jul 29 08:24:05.737132 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.737095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmfz6\" (UniqueName: \"kubernetes.io/projected/3d950279-670c-4203-94ae-46c922d2b1f6-kube-api-access-rmfz6\") pod \"feast-operator-controller-manager-7f54bd498c-zfxg6\" (UID: \"3d950279-670c-4203-94ae-46c922d2b1f6\") " pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" Jul 29 08:24:05.837799 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.837762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmfz6\" (UniqueName: \"kubernetes.io/projected/3d950279-670c-4203-94ae-46c922d2b1f6-kube-api-access-rmfz6\") pod \"feast-operator-controller-manager-7f54bd498c-zfxg6\" (UID: \"3d950279-670c-4203-94ae-46c922d2b1f6\") " pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" Jul 29 08:24:05.847291 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.847259 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmfz6\" (UniqueName: \"kubernetes.io/projected/3d950279-670c-4203-94ae-46c922d2b1f6-kube-api-access-rmfz6\") pod \"feast-operator-controller-manager-7f54bd498c-zfxg6\" (UID: \"3d950279-670c-4203-94ae-46c922d2b1f6\") " pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" Jul 29 08:24:05.957007 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:05.956974 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" Jul 29 08:24:06.088992 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:06.088962 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6"] Jul 29 08:24:06.092554 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:24:06.092528 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d950279_670c_4203_94ae_46c922d2b1f6.slice/crio-02505fd70d620b3738f60380f0344a5592a4ce4813aed34cbd8ffa173eb4de26 WatchSource:0}: Error finding container 02505fd70d620b3738f60380f0344a5592a4ce4813aed34cbd8ffa173eb4de26: Status 404 returned error can't find the container with id 02505fd70d620b3738f60380f0344a5592a4ce4813aed34cbd8ffa173eb4de26 Jul 29 08:24:06.094500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:06.094485 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:24:06.998137 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:06.998096 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" event={"ID":"3d950279-670c-4203-94ae-46c922d2b1f6","Type":"ContainerStarted","Data":"02505fd70d620b3738f60380f0344a5592a4ce4813aed34cbd8ffa173eb4de26"} Jul 29 08:24:09.007907 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:09.007871 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" event={"ID":"3d950279-670c-4203-94ae-46c922d2b1f6","Type":"ContainerStarted","Data":"92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba"} Jul 29 08:24:09.008318 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:09.007963 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" Jul 29 08:24:09.024598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:09.024552 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" podStartSLOduration=1.6277070280000001 podStartE2EDuration="4.024537223s" totalImagesPullingTime="2.396830195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:24:09.023433299 +0000 UTC m=+369.336614159" watchObservedRunningTime="2026-07-29 08:24:09.024537223 +0000 UTC m=+369.337718083" Jul 29 08:24:20.014212 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:20.014178 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" Jul 29 08:24:42.735012 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:42.734960 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:42.829287 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:42.829238 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:43.028838 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.028752 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:43.129799 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.129762 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:43.201984 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.201952 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:43.258195 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.258162 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx"] Jul 29 08:24:43.263167 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.263145 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.265932 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.265911 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-bx9xd\"" Jul 29 08:24:43.266137 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.266111 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 29 08:24:43.266263 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.266196 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 29 08:24:43.266414 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.266397 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 29 08:24:43.266502 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.266414 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 29 08:24:43.266502 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.266443 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 29 08:24:43.266502 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.266483 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 29 08:24:43.266801 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.266524 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 29 08:24:43.274496 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.274473 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx"] Jul 29 08:24:43.353717 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.353609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-offline-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.353717 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.353681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-online-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.353948 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.353820 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-registry-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.353948 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.353850 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39126b00-feaa-4a07-b85d-6470dde5d897-feast-data\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.353948 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.353878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2nm\" (UniqueName: \"kubernetes.io/projected/39126b00-feaa-4a07-b85d-6470dde5d897-kube-api-access-pn2nm\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.353948 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.353926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-ui-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.397838 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.397812 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:43.455107 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.455076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-ui-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.455272 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.455124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-offline-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.455272 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.455142 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-online-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.455272 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.455198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-registry-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.455272 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.455215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39126b00-feaa-4a07-b85d-6470dde5d897-feast-data\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.455272 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.455231 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2nm\" (UniqueName: \"kubernetes.io/projected/39126b00-feaa-4a07-b85d-6470dde5d897-kube-api-access-pn2nm\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.455704 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.455678 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39126b00-feaa-4a07-b85d-6470dde5d897-feast-data\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.457826 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.457805 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-ui-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.458037 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.458016 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-registry-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.458089 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.458016 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-online-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.458131 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.458081 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-offline-tls\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.463981 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.463958 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2nm\" (UniqueName: \"kubernetes.io/projected/39126b00-feaa-4a07-b85d-6470dde5d897-kube-api-access-pn2nm\") pod \"feast-simple-feast-setup-6b748c48cb-6dpgx\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.498580 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.498551 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:24:43.585893 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.585862 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:24:43.729082 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:43.729061 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx"] Jul 29 08:24:43.731375 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:24:43.731350 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39126b00_feaa_4a07_b85d_6470dde5d897.slice/crio-e634ae9f6b92c991caa411609990c43a3291ac486dccb6a3105812db68a2b276 WatchSource:0}: Error finding container e634ae9f6b92c991caa411609990c43a3291ac486dccb6a3105812db68a2b276: Status 404 returned error can't find the container with id e634ae9f6b92c991caa411609990c43a3291ac486dccb6a3105812db68a2b276 Jul 29 08:24:44.149931 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:24:44.149890 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerStarted","Data":"e634ae9f6b92c991caa411609990c43a3291ac486dccb6a3105812db68a2b276"} Jul 29 08:25:10.267308 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:10.267276 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerStarted","Data":"9524cc7048bfb0a32c57c864dfcdf1f1f684c369ac204abde54f452d019d67af"} Jul 29 08:25:14.285991 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:14.285908 2582 generic.go:356] "Generic (PLEG): container finished" podID="39126b00-feaa-4a07-b85d-6470dde5d897" containerID="9524cc7048bfb0a32c57c864dfcdf1f1f684c369ac204abde54f452d019d67af" exitCode=0 Jul 29 08:25:14.285991 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:14.285980 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerDied","Data":"9524cc7048bfb0a32c57c864dfcdf1f1f684c369ac204abde54f452d019d67af"} Jul 29 08:25:15.291550 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:15.291517 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerStarted","Data":"75f51d0ffd670bae88d88768360b49fc23cfdf9a12ac066486078f5b5b9072be"} Jul 29 08:25:20.313028 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:20.312980 2582 generic.go:356] "Generic (PLEG): container finished" podID="39126b00-feaa-4a07-b85d-6470dde5d897" containerID="75f51d0ffd670bae88d88768360b49fc23cfdf9a12ac066486078f5b5b9072be" exitCode=0 Jul 29 08:25:20.313405 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:20.313051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerDied","Data":"75f51d0ffd670bae88d88768360b49fc23cfdf9a12ac066486078f5b5b9072be"} Jul 29 08:25:20.322613 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:20.322592 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 29 08:25:21.322865 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:21.322789 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerStarted","Data":"ab189a66228aabbc1044a86414793a8d70445e865e9a7680f7a17a28bb0e619b"} Jul 29 08:25:21.322865 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:21.322844 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerStarted","Data":"059619c353441e9e21c1aa6ea76150d3a2b6e2864e7599d612e2629d28b275bc"} Jul 29 08:25:21.322865 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:21.322864 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerStarted","Data":"518ce3089937e185c2c4cf9254a8dc10543e2b2e7eb14f35bd4862725b8f5e85"} Jul 29 08:25:21.322865 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:21.322882 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerStarted","Data":"5e2e2cd1e4f9ab162c3b8b697a756e19c05046bdea0a0c5c3afe044ec50946a7"} Jul 29 08:25:21.351234 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:21.351161 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podStartSLOduration=12.852267382 podStartE2EDuration="38.351139764s" totalImagesPullingTime="25.498872382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:25:21.348837191 +0000 UTC m=+441.662018052" watchObservedRunningTime="2026-07-29 08:25:21.351139764 +0000 UTC m=+441.664320627" Jul 29 08:25:22.586929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:22.586806 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:22.586929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:22.586869 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:22.586929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:22.586889 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:22.586929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:22.586906 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:22.588922 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:22.588887 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Jul 29 08:25:22.588922 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:22.588915 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Jul 29 08:25:22.589137 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:22.588910 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 29 08:25:22.589137 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:22.589009 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 29 08:25:23.586411 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:23.586359 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:23.586673 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:23.586449 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:23.586673 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:23.586500 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:23.586673 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:23.586522 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:25.587045 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:25.587003 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 29 08:25:25.587045 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:25.587058 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Jul 29 08:25:25.587603 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:25.586994 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 29 08:25:25.587603 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:25.587527 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Jul 29 08:25:28.587319 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:28.587286 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:28.587802 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:28.587437 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:28.587802 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:28.587676 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:28.588273 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:28.588249 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:28.588383 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:28.588306 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:28.588486 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:28.588475 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:28.592441 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:28.592424 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:29.368767 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:29.368733 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:25:29.768610 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:25:29.768578 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:29.244934 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:29.244897 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:30.576569 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.576523 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:30.595313 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.595283 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx"] Jul 29 08:26:30.595555 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.595523 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="registry" containerID="cri-o://5e2e2cd1e4f9ab162c3b8b697a756e19c05046bdea0a0c5c3afe044ec50946a7" gracePeriod=30 Jul 29 08:26:30.595730 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.595648 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="ui" containerID="cri-o://ab189a66228aabbc1044a86414793a8d70445e865e9a7680f7a17a28bb0e619b" gracePeriod=30 Jul 29 08:26:30.595730 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.595672 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="offline" containerID="cri-o://059619c353441e9e21c1aa6ea76150d3a2b6e2864e7599d612e2629d28b275bc" gracePeriod=30 Jul 29 08:26:30.595852 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.595673 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="online" containerID="cri-o://518ce3089937e185c2c4cf9254a8dc10543e2b2e7eb14f35bd4862725b8f5e85" gracePeriod=30 Jul 29 08:26:30.676735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.676693 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:30.776918 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.776887 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:30.877149 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.877053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:30.884255 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.884224 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb"] Jul 29 08:26:30.887409 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.887387 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:30.889613 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.889589 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-kbj6x\"" Jul 29 08:26:30.901171 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.901142 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb"] Jul 29 08:26:30.976886 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.976845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-offline-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:30.977080 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.976919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-registry-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:30.977080 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.976948 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-online-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:30.977080 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.976980 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/289077e6-b350-4895-9d30-2d4065583cb6-feast-data\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:30.977080 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.977042 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-ui-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:30.977080 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.977058 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v299t\" (UniqueName: \"kubernetes.io/projected/289077e6-b350-4895-9d30-2d4065583cb6-kube-api-access-v299t\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:30.977750 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:30.977723 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:31.077940 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.077896 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-ui-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.077940 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.077933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v299t\" (UniqueName: \"kubernetes.io/projected/289077e6-b350-4895-9d30-2d4065583cb6-kube-api-access-v299t\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.078220 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.077997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-offline-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.078220 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.078037 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-registry-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.078220 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:26:31.078051 2582 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 29 08:26:31.078220 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:26:31.078115 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-ui-tls podName:289077e6-b350-4895-9d30-2d4065583cb6 nodeName:}" failed. No retries permitted until 2026-07-29 08:26:31.578094712 +0000 UTC m=+511.891275552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-ui-tls") pod "feast-simple-feast-setup-6b748c48cb-xbqlb" (UID: "289077e6-b350-4895-9d30-2d4065583cb6") : secret "feast-simple-feast-setup-ui-tls" not found Jul 29 08:26:31.078432 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.078236 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-online-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.078432 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.078285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/289077e6-b350-4895-9d30-2d4065583cb6-feast-data\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.078703 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.078678 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/289077e6-b350-4895-9d30-2d4065583cb6-feast-data\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.079295 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.079263 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:31.080812 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.080790 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-offline-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.081351 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.081330 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-registry-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.081431 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.081330 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-online-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.086232 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.086200 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v299t\" (UniqueName: \"kubernetes.io/projected/289077e6-b350-4895-9d30-2d4065583cb6-kube-api-access-v299t\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.179141 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.179112 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:31.279522 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.279428 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:31.378900 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.378867 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:31.582067 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.582034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-ui-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.584633 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.584598 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-ui-tls\") pod \"feast-simple-feast-setup-6b748c48cb-xbqlb\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.631463 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.631431 2582 generic.go:356] "Generic (PLEG): container finished" podID="39126b00-feaa-4a07-b85d-6470dde5d897" containerID="ab189a66228aabbc1044a86414793a8d70445e865e9a7680f7a17a28bb0e619b" exitCode=0 Jul 29 08:26:31.631463 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.631454 2582 generic.go:356] "Generic (PLEG): container finished" podID="39126b00-feaa-4a07-b85d-6470dde5d897" containerID="518ce3089937e185c2c4cf9254a8dc10543e2b2e7eb14f35bd4862725b8f5e85" exitCode=0 Jul 29 08:26:31.631685 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.631505 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerDied","Data":"ab189a66228aabbc1044a86414793a8d70445e865e9a7680f7a17a28bb0e619b"} Jul 29 08:26:31.631685 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.631545 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerDied","Data":"518ce3089937e185c2c4cf9254a8dc10543e2b2e7eb14f35bd4862725b8f5e85"} Jul 29 08:26:31.796221 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.796180 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:31.934790 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:31.934767 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb"] Jul 29 08:26:31.936926 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:26:31.936902 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod289077e6_b350_4895_9d30_2d4065583cb6.slice/crio-d164fcf8ff4e74e76c3174bc96a68d3fd4ecb1d66614d394d475b3a13460ba78 WatchSource:0}: Error finding container d164fcf8ff4e74e76c3174bc96a68d3fd4ecb1d66614d394d475b3a13460ba78: Status 404 returned error can't find the container with id d164fcf8ff4e74e76c3174bc96a68d3fd4ecb1d66614d394d475b3a13460ba78 Jul 29 08:26:32.637796 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:32.637758 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerStarted","Data":"9e050672b1d7393d31a3902344c5c62a0a911944dea12521c4e8986df69f012c"} Jul 29 08:26:32.637796 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:32.637804 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerStarted","Data":"d164fcf8ff4e74e76c3174bc96a68d3fd4ecb1d66614d394d475b3a13460ba78"} Jul 29 08:26:37.661984 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:37.661948 2582 generic.go:356] "Generic (PLEG): container finished" podID="289077e6-b350-4895-9d30-2d4065583cb6" containerID="9e050672b1d7393d31a3902344c5c62a0a911944dea12521c4e8986df69f012c" exitCode=0 Jul 29 08:26:37.662386 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:37.662021 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerDied","Data":"9e050672b1d7393d31a3902344c5c62a0a911944dea12521c4e8986df69f012c"} Jul 29 08:26:38.588435 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:38.588379 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 29 08:26:38.668021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:38.667985 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerStarted","Data":"6218d6780b49e0d5bcca5060b41cb6c94f39d0081fc05f06c8f5442fc42ee6aa"} Jul 29 08:26:39.366528 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:39.366478 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 29 08:26:43.692649 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:43.692546 2582 generic.go:356] "Generic (PLEG): container finished" podID="289077e6-b350-4895-9d30-2d4065583cb6" containerID="6218d6780b49e0d5bcca5060b41cb6c94f39d0081fc05f06c8f5442fc42ee6aa" exitCode=0 Jul 29 08:26:43.693089 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:43.692640 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerDied","Data":"6218d6780b49e0d5bcca5060b41cb6c94f39d0081fc05f06c8f5442fc42ee6aa"} Jul 29 08:26:44.703333 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:44.703291 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerStarted","Data":"e7c6d70c6fc1a8928336614714c1f1d91397b826369c883351fe7d25d971b0db"} Jul 29 08:26:44.703842 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:44.703355 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerStarted","Data":"72379decf5e5dee91ada5b32d546345a476a57ab2cc588ddeae54e5b3577cc8a"} Jul 29 08:26:44.703842 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:44.703374 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerStarted","Data":"d23ae9a619c37925227178eb4a23ed977aababe16cd0598118c493f1b5e410f7"} Jul 29 08:26:44.703842 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:44.703392 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerStarted","Data":"5a57e9462301d00751456257c774ffb005cf329977efe3383784757a461b500f"} Jul 29 08:26:44.730714 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:44.730648 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podStartSLOduration=14.73060541 podStartE2EDuration="14.73060541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:26:44.730286751 +0000 UTC m=+525.043467644" watchObservedRunningTime="2026-07-29 08:26:44.73060541 +0000 UTC m=+525.043786271" Jul 29 08:26:46.796945 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:46.796901 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:46.797461 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:46.796973 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:46.797461 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:46.796993 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:46.797461 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:46.797011 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:46.799372 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:46.799342 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 29 08:26:46.799644 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:46.799336 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused" Jul 29 08:26:46.799763 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:46.799419 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Jul 29 08:26:46.799763 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:46.799426 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 29 08:26:48.588671 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:48.588610 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 29 08:26:49.366135 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:49.366085 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 29 08:26:49.797488 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:49.797446 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 29 08:26:49.798069 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:49.798045 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:49.798724 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:49.798700 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:49.798850 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:49.798745 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:49.799015 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:49.798997 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:49.803763 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:49.803745 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:50.738570 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:50.738537 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:50.738754 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:50.738584 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:50.739105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:50.739084 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:50.741879 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:50.741858 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:51.796922 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:51.796870 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:52.797600 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:52.797567 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:53.752824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:53.752792 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:26:54.092341 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:54.092252 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:26:58.588578 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:58.588533 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 29 08:26:58.589038 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:58.588706 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:26:59.366082 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:59.366037 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 29 08:26:59.366281 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:26:59.366173 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:27:00.785829 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:00.785799 2582 generic.go:356] "Generic (PLEG): container finished" podID="39126b00-feaa-4a07-b85d-6470dde5d897" containerID="059619c353441e9e21c1aa6ea76150d3a2b6e2864e7599d612e2629d28b275bc" exitCode=137 Jul 29 08:27:00.785829 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:00.785822 2582 generic.go:356] "Generic (PLEG): container finished" podID="39126b00-feaa-4a07-b85d-6470dde5d897" containerID="5e2e2cd1e4f9ab162c3b8b697a756e19c05046bdea0a0c5c3afe044ec50946a7" exitCode=137 Jul 29 08:27:00.786265 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:00.785848 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerDied","Data":"059619c353441e9e21c1aa6ea76150d3a2b6e2864e7599d612e2629d28b275bc"} Jul 29 08:27:00.786265 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:00.785880 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerDied","Data":"5e2e2cd1e4f9ab162c3b8b697a756e19c05046bdea0a0c5c3afe044ec50946a7"} Jul 29 08:27:01.252371 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.252340 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:27:01.355234 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.355192 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39126b00-feaa-4a07-b85d-6470dde5d897-feast-data\") pod \"39126b00-feaa-4a07-b85d-6470dde5d897\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " Jul 29 08:27:01.355234 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.355238 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-online-tls\") pod \"39126b00-feaa-4a07-b85d-6470dde5d897\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " Jul 29 08:27:01.355446 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.355272 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-offline-tls\") pod \"39126b00-feaa-4a07-b85d-6470dde5d897\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " Jul 29 08:27:01.355446 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.355293 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-ui-tls\") pod \"39126b00-feaa-4a07-b85d-6470dde5d897\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " Jul 29 08:27:01.355446 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.355361 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn2nm\" (UniqueName: \"kubernetes.io/projected/39126b00-feaa-4a07-b85d-6470dde5d897-kube-api-access-pn2nm\") pod \"39126b00-feaa-4a07-b85d-6470dde5d897\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " Jul 29 08:27:01.355446 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.355395 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-registry-tls\") pod \"39126b00-feaa-4a07-b85d-6470dde5d897\" (UID: \"39126b00-feaa-4a07-b85d-6470dde5d897\") " Jul 29 08:27:01.355876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.355843 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39126b00-feaa-4a07-b85d-6470dde5d897-feast-data" (OuterVolumeSpecName: "feast-data") pod "39126b00-feaa-4a07-b85d-6470dde5d897" (UID: "39126b00-feaa-4a07-b85d-6470dde5d897"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:27:01.357744 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.357713 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "39126b00-feaa-4a07-b85d-6470dde5d897" (UID: "39126b00-feaa-4a07-b85d-6470dde5d897"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:27:01.358247 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.358220 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-online-tls" (OuterVolumeSpecName: "online-tls") pod "39126b00-feaa-4a07-b85d-6470dde5d897" (UID: "39126b00-feaa-4a07-b85d-6470dde5d897"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:27:01.358247 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.358235 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "39126b00-feaa-4a07-b85d-6470dde5d897" (UID: "39126b00-feaa-4a07-b85d-6470dde5d897"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:27:01.358366 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.358262 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "39126b00-feaa-4a07-b85d-6470dde5d897" (UID: "39126b00-feaa-4a07-b85d-6470dde5d897"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:27:01.358366 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.358288 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39126b00-feaa-4a07-b85d-6470dde5d897-kube-api-access-pn2nm" (OuterVolumeSpecName: "kube-api-access-pn2nm") pod "39126b00-feaa-4a07-b85d-6470dde5d897" (UID: "39126b00-feaa-4a07-b85d-6470dde5d897"). InnerVolumeSpecName "kube-api-access-pn2nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:27:01.456051 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.456009 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn2nm\" (UniqueName: \"kubernetes.io/projected/39126b00-feaa-4a07-b85d-6470dde5d897-kube-api-access-pn2nm\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:27:01.456051 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.456042 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:27:01.456051 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.456055 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39126b00-feaa-4a07-b85d-6470dde5d897-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:27:01.456292 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.456066 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:27:01.456292 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.456077 2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-offline-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:27:01.456292 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.456087 2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39126b00-feaa-4a07-b85d-6470dde5d897-ui-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:27:01.794238 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.794198 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" event={"ID":"39126b00-feaa-4a07-b85d-6470dde5d897","Type":"ContainerDied","Data":"e634ae9f6b92c991caa411609990c43a3291ac486dccb6a3105812db68a2b276"} Jul 29 08:27:01.794730 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.794255 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx" Jul 29 08:27:01.794730 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.794267 2582 scope.go:117] "RemoveContainer" containerID="ab189a66228aabbc1044a86414793a8d70445e865e9a7680f7a17a28bb0e619b" Jul 29 08:27:01.804454 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.804433 2582 scope.go:117] "RemoveContainer" containerID="059619c353441e9e21c1aa6ea76150d3a2b6e2864e7599d612e2629d28b275bc" Jul 29 08:27:01.812688 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.812666 2582 scope.go:117] "RemoveContainer" containerID="518ce3089937e185c2c4cf9254a8dc10543e2b2e7eb14f35bd4862725b8f5e85" Jul 29 08:27:01.821365 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.821336 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx"] Jul 29 08:27:01.822093 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.822071 2582 scope.go:117] "RemoveContainer" containerID="5e2e2cd1e4f9ab162c3b8b697a756e19c05046bdea0a0c5c3afe044ec50946a7" Jul 29 08:27:01.829941 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.829920 2582 scope.go:117] "RemoveContainer" containerID="75f51d0ffd670bae88d88768360b49fc23cfdf9a12ac066486078f5b5b9072be" Jul 29 08:27:01.833019 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.832999 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-6dpgx"] Jul 29 08:27:01.838583 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:01.838566 2582 scope.go:117] "RemoveContainer" containerID="9524cc7048bfb0a32c57c864dfcdf1f1f684c369ac204abde54f452d019d67af" Jul 29 08:27:02.250429 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:02.250396 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" path="/var/lib/kubelet/pods/39126b00-feaa-4a07-b85d-6470dde5d897/volumes" Jul 29 08:27:48.882429 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:48.882395 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:27:56.448600 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.448560 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:27:56.508565 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.508529 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:27:56.652829 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.652796 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:27:56.745845 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.745808 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:27:56.750429 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750400 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb"] Jul 29 08:27:56.750808 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750792 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="feast-apply" Jul 29 08:27:56.750905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750809 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="feast-apply" Jul 29 08:27:56.750905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750823 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="registry" Jul 29 08:27:56.750905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750830 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="registry" Jul 29 08:27:56.750905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750850 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="feast-init" Jul 29 08:27:56.750905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750858 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="feast-init" Jul 29 08:27:56.750905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750868 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="online" Jul 29 08:27:56.750905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750876 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="online" Jul 29 08:27:56.750905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750890 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="ui" Jul 29 08:27:56.750905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750898 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="ui" Jul 29 08:27:56.750905 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750908 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="offline" Jul 29 08:27:56.751421 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.750916 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="offline" Jul 29 08:27:56.751421 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.751006 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="ui" Jul 29 08:27:56.751421 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.751021 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="online" Jul 29 08:27:56.751421 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.751035 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="registry" Jul 29 08:27:56.751421 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.751043 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="39126b00-feaa-4a07-b85d-6470dde5d897" containerName="offline" Jul 29 08:27:56.754793 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.754774 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.757879 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.757853 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 29 08:27:56.758041 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.757862 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 29 08:27:56.758126 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.757862 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 29 08:27:56.758126 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.757874 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 29 08:27:56.758126 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.757874 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 29 08:27:56.758281 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.757877 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 29 08:27:56.758281 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.758070 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 29 08:27:56.758449 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.758435 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-jpbjm\"" Jul 29 08:27:56.768275 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.768252 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb"] Jul 29 08:27:56.834997 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.834957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/322fb4f6-c363-4b54-b12a-e9939de333af-registry-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.834997 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.835001 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/322fb4f6-c363-4b54-b12a-e9939de333af-feast-data\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.835214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.835065 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-ui-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.835214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.835106 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4wg\" (UniqueName: \"kubernetes.io/projected/322fb4f6-c363-4b54-b12a-e9939de333af-kube-api-access-gd4wg\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.835214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.835172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-online-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.835214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.835204 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-offline-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.935993 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.935957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/322fb4f6-c363-4b54-b12a-e9939de333af-registry-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.935993 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.935996 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/322fb4f6-c363-4b54-b12a-e9939de333af-feast-data\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.936247 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.936148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-ui-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.936247 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.936196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4wg\" (UniqueName: \"kubernetes.io/projected/322fb4f6-c363-4b54-b12a-e9939de333af-kube-api-access-gd4wg\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.936247 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.936237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-online-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.936403 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.936290 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-offline-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.936403 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:27:56.936304 2582 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 29 08:27:56.936403 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:27:56.936383 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-ui-tls podName:322fb4f6-c363-4b54-b12a-e9939de333af nodeName:}" failed. No retries permitted until 2026-07-29 08:27:57.436360293 +0000 UTC m=+597.749541134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-ui-tls") pod "feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" (UID: "322fb4f6-c363-4b54-b12a-e9939de333af") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 29 08:27:56.936403 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.936398 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/322fb4f6-c363-4b54-b12a-e9939de333af-feast-data\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.936603 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:27:56.936435 2582 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 29 08:27:56.936603 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:27:56.936496 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-offline-tls podName:322fb4f6-c363-4b54-b12a-e9939de333af nodeName:}" failed. No retries permitted until 2026-07-29 08:27:57.436479818 +0000 UTC m=+597.749660673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-offline-tls") pod "feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" (UID: "322fb4f6-c363-4b54-b12a-e9939de333af") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 29 08:27:56.936760 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.936741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/322fb4f6-c363-4b54-b12a-e9939de333af-registry-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.937456 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.937430 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:27:56.939132 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.939111 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-online-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:56.945909 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:56.945886 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4wg\" (UniqueName: \"kubernetes.io/projected/322fb4f6-c363-4b54-b12a-e9939de333af-kube-api-access-gd4wg\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:57.037951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:57.037868 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:27:57.138282 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:57.138228 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:27:57.440655 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:57.440563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-ui-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:57.440655 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:57.440634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-offline-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:57.443195 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:57.443170 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-offline-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:57.443338 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:57.443221 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-ui-tls\") pod \"feast-simple-feast-remote-setup-764c6bbd9f-hkqqb\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:57.663507 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:57.663476 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:27:57.799459 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:57.799434 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb"] Jul 29 08:27:57.801236 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:27:57.801209 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod322fb4f6_c363_4b54_b12a_e9939de333af.slice/crio-923eff93ea3d2f71e3ae0daad3ded876e87aabadd6563b9ce11df6ab5c48e62b WatchSource:0}: Error finding container 923eff93ea3d2f71e3ae0daad3ded876e87aabadd6563b9ce11df6ab5c48e62b: Status 404 returned error can't find the container with id 923eff93ea3d2f71e3ae0daad3ded876e87aabadd6563b9ce11df6ab5c48e62b Jul 29 08:27:58.027286 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:58.027205 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerStarted","Data":"15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f"} Jul 29 08:27:58.027286 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:58.027249 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerStarted","Data":"923eff93ea3d2f71e3ae0daad3ded876e87aabadd6563b9ce11df6ab5c48e62b"} Jul 29 08:27:58.479680 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:27:58.479639 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:28:03.050776 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:03.050744 2582 generic.go:356] "Generic (PLEG): container finished" podID="322fb4f6-c363-4b54-b12a-e9939de333af" containerID="15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f" exitCode=0 Jul 29 08:28:03.051174 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:03.050826 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerDied","Data":"15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f"} Jul 29 08:28:04.058273 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:04.058240 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerStarted","Data":"0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7"} Jul 29 08:28:09.080805 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:09.080771 2582 generic.go:356] "Generic (PLEG): container finished" podID="322fb4f6-c363-4b54-b12a-e9939de333af" containerID="0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7" exitCode=0 Jul 29 08:28:09.080805 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:09.080809 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerDied","Data":"0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7"} Jul 29 08:28:09.087270 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:09.087250 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 29 08:28:10.089871 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:10.089838 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerStarted","Data":"e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e"} Jul 29 08:28:10.090362 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:10.089891 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerStarted","Data":"b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1"} Jul 29 08:28:10.090362 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:10.089909 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerStarted","Data":"75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90"} Jul 29 08:28:10.116184 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:10.116120 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podStartSLOduration=14.116098256 podStartE2EDuration="14.116098256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:28:10.11297392 +0000 UTC m=+610.426154782" watchObservedRunningTime="2026-07-29 08:28:10.116098256 +0000 UTC m=+610.429279120" Jul 29 08:28:12.664700 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:12.664655 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:12.665340 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:12.664722 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:12.665340 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:12.664744 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:12.666438 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:12.666405 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Jul 29 08:28:12.666553 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:12.666473 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused" Jul 29 08:28:12.666553 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:12.666502 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 29 08:28:15.665464 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:15.665436 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:15.665904 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:15.665566 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:15.666025 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:15.666001 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:15.666164 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:15.666046 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:15.666231 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:15.666204 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:15.666427 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:15.666412 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:15.672385 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:15.672363 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:16.119414 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:16.119388 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:16.123962 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:16.123929 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:28:16.381876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:28:16.381789 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:29:11.680615 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:11.680577 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:29:18.159973 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.159930 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:29:18.176192 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.176164 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb"] Jul 29 08:29:18.176443 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.176417 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="online" containerID="cri-o://75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90" gracePeriod=30 Jul 29 08:29:18.176656 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.176589 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="offline" containerID="cri-o://b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1" gracePeriod=30 Jul 29 08:29:18.176907 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.176651 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="ui" containerID="cri-o://e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e" gracePeriod=30 Jul 29 08:29:18.260181 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.260143 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:29:18.307835 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.307805 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb"] Jul 29 08:29:18.308067 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.308045 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="registry" containerID="cri-o://5a57e9462301d00751456257c774ffb005cf329977efe3383784757a461b500f" gracePeriod=30 Jul 29 08:29:18.308168 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.308139 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="offline" containerID="cri-o://72379decf5e5dee91ada5b32d546345a476a57ab2cc588ddeae54e5b3577cc8a" gracePeriod=30 Jul 29 08:29:18.308226 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.308172 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="online" containerID="cri-o://d23ae9a619c37925227178eb4a23ed977aababe16cd0598118c493f1b5e410f7" gracePeriod=30 Jul 29 08:29:18.308226 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.308140 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="ui" containerID="cri-o://e7c6d70c6fc1a8928336614714c1f1d91397b826369c883351fe7d25d971b0db" gracePeriod=30 Jul 29 08:29:18.337021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.336991 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:29:18.402407 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:18.402364 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:29:19.387688 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:19.387593 2582 generic.go:356] "Generic (PLEG): container finished" podID="289077e6-b350-4895-9d30-2d4065583cb6" containerID="e7c6d70c6fc1a8928336614714c1f1d91397b826369c883351fe7d25d971b0db" exitCode=0 Jul 29 08:29:19.387688 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:19.387635 2582 generic.go:356] "Generic (PLEG): container finished" podID="289077e6-b350-4895-9d30-2d4065583cb6" containerID="d23ae9a619c37925227178eb4a23ed977aababe16cd0598118c493f1b5e410f7" exitCode=0 Jul 29 08:29:19.387688 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:19.387668 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerDied","Data":"e7c6d70c6fc1a8928336614714c1f1d91397b826369c883351fe7d25d971b0db"} Jul 29 08:29:19.388090 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:19.387715 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerDied","Data":"d23ae9a619c37925227178eb4a23ed977aababe16cd0598118c493f1b5e410f7"} Jul 29 08:29:19.390139 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:19.390122 2582 generic.go:356] "Generic (PLEG): container finished" podID="322fb4f6-c363-4b54-b12a-e9939de333af" containerID="e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e" exitCode=0 Jul 29 08:29:19.390198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:19.390139 2582 generic.go:356] "Generic (PLEG): container finished" podID="322fb4f6-c363-4b54-b12a-e9939de333af" containerID="75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90" exitCode=0 Jul 29 08:29:19.390198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:19.390188 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerDied","Data":"e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e"} Jul 29 08:29:19.390260 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:19.390216 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerDied","Data":"75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90"} Jul 29 08:29:20.739562 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:20.739515 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 29 08:29:23.477110 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:23.477072 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:29:23.752923 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:23.752831 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 29 08:29:25.666340 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:25.666293 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Jul 29 08:29:26.119796 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:26.119759 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 29 08:29:30.738708 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:30.738666 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 29 08:29:33.753088 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:33.753045 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 29 08:29:35.666313 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:35.666266 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Jul 29 08:29:36.120018 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:36.119974 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 29 08:29:40.739327 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:40.739278 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 29 08:29:40.739776 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:40.739437 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:29:43.752342 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:43.752298 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 29 08:29:43.752807 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:43.752410 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:29:45.666401 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:45.666355 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.30:8443: connect: connection refused" Jul 29 08:29:45.666926 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:45.666520 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:29:46.119549 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:46.119509 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 29 08:29:46.119749 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:46.119675 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:29:48.418832 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.418808 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:29:48.518294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.518261 2582 generic.go:356] "Generic (PLEG): container finished" podID="289077e6-b350-4895-9d30-2d4065583cb6" containerID="72379decf5e5dee91ada5b32d546345a476a57ab2cc588ddeae54e5b3577cc8a" exitCode=137 Jul 29 08:29:48.518294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.518283 2582 generic.go:356] "Generic (PLEG): container finished" podID="289077e6-b350-4895-9d30-2d4065583cb6" containerID="5a57e9462301d00751456257c774ffb005cf329977efe3383784757a461b500f" exitCode=137 Jul 29 08:29:48.518521 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.518332 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerDied","Data":"72379decf5e5dee91ada5b32d546345a476a57ab2cc588ddeae54e5b3577cc8a"} Jul 29 08:29:48.518521 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.518373 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerDied","Data":"5a57e9462301d00751456257c774ffb005cf329977efe3383784757a461b500f"} Jul 29 08:29:48.520720 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.520697 2582 generic.go:356] "Generic (PLEG): container finished" podID="322fb4f6-c363-4b54-b12a-e9939de333af" containerID="b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1" exitCode=137 Jul 29 08:29:48.520817 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.520750 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerDied","Data":"b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1"} Jul 29 08:29:48.520817 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.520762 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" Jul 29 08:29:48.520817 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.520786 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb" event={"ID":"322fb4f6-c363-4b54-b12a-e9939de333af","Type":"ContainerDied","Data":"923eff93ea3d2f71e3ae0daad3ded876e87aabadd6563b9ce11df6ab5c48e62b"} Jul 29 08:29:48.520817 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.520807 2582 scope.go:117] "RemoveContainer" containerID="e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e" Jul 29 08:29:48.529774 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.529759 2582 scope.go:117] "RemoveContainer" containerID="b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1" Jul 29 08:29:48.536218 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.536190 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-offline-tls\") pod \"322fb4f6-c363-4b54-b12a-e9939de333af\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " Jul 29 08:29:48.536310 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.536257 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd4wg\" (UniqueName: \"kubernetes.io/projected/322fb4f6-c363-4b54-b12a-e9939de333af-kube-api-access-gd4wg\") pod \"322fb4f6-c363-4b54-b12a-e9939de333af\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " Jul 29 08:29:48.536310 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.536290 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/322fb4f6-c363-4b54-b12a-e9939de333af-registry-tls\") pod \"322fb4f6-c363-4b54-b12a-e9939de333af\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " Jul 29 08:29:48.536413 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.536366 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/322fb4f6-c363-4b54-b12a-e9939de333af-feast-data\") pod \"322fb4f6-c363-4b54-b12a-e9939de333af\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " Jul 29 08:29:48.536413 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.536395 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-online-tls\") pod \"322fb4f6-c363-4b54-b12a-e9939de333af\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " Jul 29 08:29:48.536525 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.536434 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-ui-tls\") pod \"322fb4f6-c363-4b54-b12a-e9939de333af\" (UID: \"322fb4f6-c363-4b54-b12a-e9939de333af\") " Jul 29 08:29:48.537077 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.537038 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322fb4f6-c363-4b54-b12a-e9939de333af-feast-data" (OuterVolumeSpecName: "feast-data") pod "322fb4f6-c363-4b54-b12a-e9939de333af" (UID: "322fb4f6-c363-4b54-b12a-e9939de333af"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:29:48.537077 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.537062 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322fb4f6-c363-4b54-b12a-e9939de333af-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "322fb4f6-c363-4b54-b12a-e9939de333af" (UID: "322fb4f6-c363-4b54-b12a-e9939de333af"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:29:48.538557 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.538537 2582 scope.go:117] "RemoveContainer" containerID="75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90" Jul 29 08:29:48.539202 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.539177 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "322fb4f6-c363-4b54-b12a-e9939de333af" (UID: "322fb4f6-c363-4b54-b12a-e9939de333af"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:29:48.539464 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.539427 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-online-tls" (OuterVolumeSpecName: "online-tls") pod "322fb4f6-c363-4b54-b12a-e9939de333af" (UID: "322fb4f6-c363-4b54-b12a-e9939de333af"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:29:48.539552 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.539503 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "322fb4f6-c363-4b54-b12a-e9939de333af" (UID: "322fb4f6-c363-4b54-b12a-e9939de333af"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:29:48.539646 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.539555 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322fb4f6-c363-4b54-b12a-e9939de333af-kube-api-access-gd4wg" (OuterVolumeSpecName: "kube-api-access-gd4wg") pod "322fb4f6-c363-4b54-b12a-e9939de333af" (UID: "322fb4f6-c363-4b54-b12a-e9939de333af"). InnerVolumeSpecName "kube-api-access-gd4wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:29:48.551558 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.551534 2582 scope.go:117] "RemoveContainer" containerID="0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7" Jul 29 08:29:48.561154 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.561138 2582 scope.go:117] "RemoveContainer" containerID="15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f" Jul 29 08:29:48.568350 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.568335 2582 scope.go:117] "RemoveContainer" containerID="e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e" Jul 29 08:29:48.568639 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:29:48.568604 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e\": container with ID starting with e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e not found: ID does not exist" containerID="e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e" Jul 29 08:29:48.568693 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.568650 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e"} err="failed to get container status \"e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e\": rpc error: code = NotFound desc = could not find container \"e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e\": container with ID starting with e05e9cde1c1f20b908835d8e95a643c0c82bd3159e4e27929746deeef581622e not found: ID does not exist" Jul 29 08:29:48.568693 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.568668 2582 scope.go:117] "RemoveContainer" containerID="b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1" Jul 29 08:29:48.568889 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:29:48.568870 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1\": container with ID starting with b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1 not found: ID does not exist" containerID="b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1" Jul 29 08:29:48.568929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.568898 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1"} err="failed to get container status \"b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1\": rpc error: code = NotFound desc = could not find container \"b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1\": container with ID starting with b94174cb079ee482ecf5c5a811393a59bb9cbb65333c8288d184cf45099080e1 not found: ID does not exist" Jul 29 08:29:48.568929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.568915 2582 scope.go:117] "RemoveContainer" containerID="75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90" Jul 29 08:29:48.569163 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:29:48.569147 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90\": container with ID starting with 75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90 not found: ID does not exist" containerID="75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90" Jul 29 08:29:48.569200 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.569169 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90"} err="failed to get container status \"75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90\": rpc error: code = NotFound desc = could not find container \"75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90\": container with ID starting with 75301cc4f35110c95064ba8ad1791efd668d1b4b2e556973df0d7ddabf485c90 not found: ID does not exist" Jul 29 08:29:48.569200 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.569182 2582 scope.go:117] "RemoveContainer" containerID="0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7" Jul 29 08:29:48.569404 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:29:48.569388 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7\": container with ID starting with 0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7 not found: ID does not exist" containerID="0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7" Jul 29 08:29:48.569444 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.569410 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7"} err="failed to get container status \"0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7\": rpc error: code = NotFound desc = could not find container \"0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7\": container with ID starting with 0db82f3f3a999678abe1cd5548143164c4fa9a8876e1f53da9956089574222c7 not found: ID does not exist" Jul 29 08:29:48.569444 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.569424 2582 scope.go:117] "RemoveContainer" containerID="15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f" Jul 29 08:29:48.569661 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:29:48.569646 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f\": container with ID starting with 15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f not found: ID does not exist" containerID="15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f" Jul 29 08:29:48.569720 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.569663 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f"} err="failed to get container status \"15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f\": rpc error: code = NotFound desc = could not find container \"15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f\": container with ID starting with 15b75b03228044116cf1e7fd46495e2989eaa303e40f24271cf90a17aa991a5f not found: ID does not exist" Jul 29 08:29:48.637426 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.637395 2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-offline-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:48.637426 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.637421 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4wg\" (UniqueName: \"kubernetes.io/projected/322fb4f6-c363-4b54-b12a-e9939de333af-kube-api-access-gd4wg\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:48.637426 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.637432 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/322fb4f6-c363-4b54-b12a-e9939de333af-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:48.637655 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.637440 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/322fb4f6-c363-4b54-b12a-e9939de333af-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:48.637655 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.637451 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:48.637655 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.637458 2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/322fb4f6-c363-4b54-b12a-e9939de333af-ui-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:48.844041 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.844010 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb"] Jul 29 08:29:48.847495 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.847473 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-764c6bbd9f-hkqqb"] Jul 29 08:29:48.976170 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:48.976146 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:29:49.040863 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.040830 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-registry-tls\") pod \"289077e6-b350-4895-9d30-2d4065583cb6\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " Jul 29 08:29:49.041151 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.040876 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-online-tls\") pod \"289077e6-b350-4895-9d30-2d4065583cb6\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " Jul 29 08:29:49.041151 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.040901 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-ui-tls\") pod \"289077e6-b350-4895-9d30-2d4065583cb6\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " Jul 29 08:29:49.041151 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.040933 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/289077e6-b350-4895-9d30-2d4065583cb6-feast-data\") pod \"289077e6-b350-4895-9d30-2d4065583cb6\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " Jul 29 08:29:49.041151 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.040981 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-offline-tls\") pod \"289077e6-b350-4895-9d30-2d4065583cb6\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " Jul 29 08:29:49.041151 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.041053 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v299t\" (UniqueName: \"kubernetes.io/projected/289077e6-b350-4895-9d30-2d4065583cb6-kube-api-access-v299t\") pod \"289077e6-b350-4895-9d30-2d4065583cb6\" (UID: \"289077e6-b350-4895-9d30-2d4065583cb6\") " Jul 29 08:29:49.041554 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.041525 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289077e6-b350-4895-9d30-2d4065583cb6-feast-data" (OuterVolumeSpecName: "feast-data") pod "289077e6-b350-4895-9d30-2d4065583cb6" (UID: "289077e6-b350-4895-9d30-2d4065583cb6"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:29:49.043164 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.043144 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "289077e6-b350-4895-9d30-2d4065583cb6" (UID: "289077e6-b350-4895-9d30-2d4065583cb6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:29:49.043749 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.043728 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "289077e6-b350-4895-9d30-2d4065583cb6" (UID: "289077e6-b350-4895-9d30-2d4065583cb6"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:29:49.043832 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.043742 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-online-tls" (OuterVolumeSpecName: "online-tls") pod "289077e6-b350-4895-9d30-2d4065583cb6" (UID: "289077e6-b350-4895-9d30-2d4065583cb6"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:29:49.043832 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.043781 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "289077e6-b350-4895-9d30-2d4065583cb6" (UID: "289077e6-b350-4895-9d30-2d4065583cb6"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:29:49.043832 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.043814 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289077e6-b350-4895-9d30-2d4065583cb6-kube-api-access-v299t" (OuterVolumeSpecName: "kube-api-access-v299t") pod "289077e6-b350-4895-9d30-2d4065583cb6" (UID: "289077e6-b350-4895-9d30-2d4065583cb6"). InnerVolumeSpecName "kube-api-access-v299t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:29:49.142599 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.142515 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v299t\" (UniqueName: \"kubernetes.io/projected/289077e6-b350-4895-9d30-2d4065583cb6-kube-api-access-v299t\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:49.142599 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.142546 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:49.142599 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.142556 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:49.142599 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.142564 2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-ui-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:49.142599 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.142572 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/289077e6-b350-4895-9d30-2d4065583cb6-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:49.142599 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.142579 2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/289077e6-b350-4895-9d30-2d4065583cb6-offline-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:29:49.528065 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.528033 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" event={"ID":"289077e6-b350-4895-9d30-2d4065583cb6","Type":"ContainerDied","Data":"d164fcf8ff4e74e76c3174bc96a68d3fd4ecb1d66614d394d475b3a13460ba78"} Jul 29 08:29:49.528518 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.528046 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb" Jul 29 08:29:49.528518 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.528082 2582 scope.go:117] "RemoveContainer" containerID="e7c6d70c6fc1a8928336614714c1f1d91397b826369c883351fe7d25d971b0db" Jul 29 08:29:49.538312 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.538288 2582 scope.go:117] "RemoveContainer" containerID="72379decf5e5dee91ada5b32d546345a476a57ab2cc588ddeae54e5b3577cc8a" Jul 29 08:29:49.546811 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.546792 2582 scope.go:117] "RemoveContainer" containerID="d23ae9a619c37925227178eb4a23ed977aababe16cd0598118c493f1b5e410f7" Jul 29 08:29:49.551264 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.551240 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb"] Jul 29 08:29:49.556064 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.556040 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6b748c48cb-xbqlb"] Jul 29 08:29:49.556475 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.556454 2582 scope.go:117] "RemoveContainer" containerID="5a57e9462301d00751456257c774ffb005cf329977efe3383784757a461b500f" Jul 29 08:29:49.564134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.564118 2582 scope.go:117] "RemoveContainer" containerID="6218d6780b49e0d5bcca5060b41cb6c94f39d0081fc05f06c8f5442fc42ee6aa" Jul 29 08:29:49.571281 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:49.571265 2582 scope.go:117] "RemoveContainer" containerID="9e050672b1d7393d31a3902344c5c62a0a911944dea12521c4e8986df69f012c" Jul 29 08:29:50.251053 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:50.250973 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="289077e6-b350-4895-9d30-2d4065583cb6" path="/var/lib/kubelet/pods/289077e6-b350-4895-9d30-2d4065583cb6/volumes" Jul 29 08:29:50.251558 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:50.251545 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" path="/var/lib/kubelet/pods/322fb4f6-c363-4b54-b12a-e9939de333af/volumes" Jul 29 08:29:57.416739 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:57.416698 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:29:57.478836 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:29:57.478793 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:30:01.273098 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.273024 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:30:01.518533 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518501 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6458"] Jul 29 08:30:01.518885 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518871 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="registry" Jul 29 08:30:01.518932 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518888 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="registry" Jul 29 08:30:01.518932 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518897 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="offline" Jul 29 08:30:01.518932 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518903 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="offline" Jul 29 08:30:01.518932 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518909 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="ui" Jul 29 08:30:01.518932 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518915 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="ui" Jul 29 08:30:01.518932 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518925 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="offline" Jul 29 08:30:01.518932 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518930 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="offline" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518941 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="ui" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518948 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="ui" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518955 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="feast-init" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518960 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="feast-init" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518965 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="feast-apply" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518970 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="feast-apply" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518975 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="online" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518980 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="online" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518988 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="feast-apply" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.518992 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="feast-apply" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519003 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="online" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519007 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="online" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519024 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="feast-init" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519029 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="feast-init" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519076 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="online" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519083 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="online" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519091 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="ui" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519101 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="ui" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519108 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="registry" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519114 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="289077e6-b350-4895-9d30-2d4065583cb6" containerName="offline" Jul 29 08:30:01.519214 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.519119 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="322fb4f6-c363-4b54-b12a-e9939de333af" containerName="offline" Jul 29 08:30:01.523576 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.523520 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-w6458" Jul 29 08:30:01.526286 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.526263 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 29 08:30:01.526394 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.526301 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-kwrcj\"" Jul 29 08:30:01.526394 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.526267 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jul 29 08:30:01.526907 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.526889 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 29 08:30:01.530567 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.530544 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6458"] Jul 29 08:30:01.562054 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.561966 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:30:01.647647 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.647587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/614f25cf-0c52-4cb2-8455-1b12c78b88e5-postgresdata\") pod \"postgres-864846d4f5-w6458\" (UID: \"614f25cf-0c52-4cb2-8455-1b12c78b88e5\") " pod="test-ns-feast/postgres-864846d4f5-w6458" Jul 29 08:30:01.647865 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.647703 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb55w\" (UniqueName: \"kubernetes.io/projected/614f25cf-0c52-4cb2-8455-1b12c78b88e5-kube-api-access-mb55w\") pod \"postgres-864846d4f5-w6458\" (UID: \"614f25cf-0c52-4cb2-8455-1b12c78b88e5\") " pod="test-ns-feast/postgres-864846d4f5-w6458" Jul 29 08:30:01.648407 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.648370 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:30:01.748172 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.748131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb55w\" (UniqueName: \"kubernetes.io/projected/614f25cf-0c52-4cb2-8455-1b12c78b88e5-kube-api-access-mb55w\") pod \"postgres-864846d4f5-w6458\" (UID: \"614f25cf-0c52-4cb2-8455-1b12c78b88e5\") " pod="test-ns-feast/postgres-864846d4f5-w6458" Jul 29 08:30:01.748359 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.748214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/614f25cf-0c52-4cb2-8455-1b12c78b88e5-postgresdata\") pod \"postgres-864846d4f5-w6458\" (UID: \"614f25cf-0c52-4cb2-8455-1b12c78b88e5\") " pod="test-ns-feast/postgres-864846d4f5-w6458" Jul 29 08:30:01.748561 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.748544 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/614f25cf-0c52-4cb2-8455-1b12c78b88e5-postgresdata\") pod \"postgres-864846d4f5-w6458\" (UID: \"614f25cf-0c52-4cb2-8455-1b12c78b88e5\") " pod="test-ns-feast/postgres-864846d4f5-w6458" Jul 29 08:30:01.757582 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.757560 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb55w\" (UniqueName: \"kubernetes.io/projected/614f25cf-0c52-4cb2-8455-1b12c78b88e5-kube-api-access-mb55w\") pod \"postgres-864846d4f5-w6458\" (UID: \"614f25cf-0c52-4cb2-8455-1b12c78b88e5\") " pod="test-ns-feast/postgres-864846d4f5-w6458" Jul 29 08:30:01.843803 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.843714 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-w6458" Jul 29 08:30:01.985534 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.985483 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6458"] Jul 29 08:30:01.988398 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:30:01.988370 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod614f25cf_0c52_4cb2_8455_1b12c78b88e5.slice/crio-114c31cd406b9a669898020c957978eeb181300aa0a6cbe8b4865b307e48fd3d WatchSource:0}: Error finding container 114c31cd406b9a669898020c957978eeb181300aa0a6cbe8b4865b307e48fd3d: Status 404 returned error can't find the container with id 114c31cd406b9a669898020c957978eeb181300aa0a6cbe8b4865b307e48fd3d Jul 29 08:30:01.990341 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:01.990321 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:30:02.582350 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:02.582319 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-w6458" event={"ID":"614f25cf-0c52-4cb2-8455-1b12c78b88e5","Type":"ContainerStarted","Data":"114c31cd406b9a669898020c957978eeb181300aa0a6cbe8b4865b307e48fd3d"} Jul 29 08:30:07.606278 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:07.606242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-w6458" event={"ID":"614f25cf-0c52-4cb2-8455-1b12c78b88e5","Type":"ContainerStarted","Data":"6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19"} Jul 29 08:30:07.622209 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:07.622137 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-w6458" podStartSLOduration=1.122528499 podStartE2EDuration="6.622117912s" totalImagesPullingTime="5.499589413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:30:07.621839784 +0000 UTC m=+727.935020644" watchObservedRunningTime="2026-07-29 08:30:07.622117912 +0000 UTC m=+727.935298776" Jul 29 08:30:10.191047 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.191013 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:30:10.278901 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.278865 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:30:10.335392 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.335345 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g"] Jul 29 08:30:10.343780 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.343759 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.346312 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.346276 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jul 29 08:30:10.346312 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.346300 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jul 29 08:30:10.346516 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.346390 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-5ts4t\"" Jul 29 08:30:10.346636 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.346604 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jul 29 08:30:10.346636 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.346633 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jul 29 08:30:10.354783 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.354761 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g"] Jul 29 08:30:10.425972 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.425941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-offline-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.425972 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.425985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt58j\" (UniqueName: \"kubernetes.io/projected/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-kube-api-access-dt58j\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.426183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.426006 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-registry-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.426183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.426052 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-online-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.426183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.426078 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-feast-data\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.526861 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.526823 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-offline-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.527048 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.526874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt58j\" (UniqueName: \"kubernetes.io/projected/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-kube-api-access-dt58j\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.527048 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.526902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-registry-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.527048 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.526924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-online-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.527048 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.526944 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-feast-data\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.527048 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:30:10.526980 2582 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Jul 29 08:30:10.527259 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:30:10.527068 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-offline-tls podName:fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2 nodeName:}" failed. No retries permitted until 2026-07-29 08:30:11.027047764 +0000 UTC m=+731.340228607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-offline-tls") pod "feast-credit-scoring-86f64d7cd-rbj6g" (UID: "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2") : secret "feast-credit-scoring-offline-tls" not found Jul 29 08:30:10.527259 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:30:10.527096 2582 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Jul 29 08:30:10.527259 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:30:10.527166 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-online-tls podName:fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2 nodeName:}" failed. No retries permitted until 2026-07-29 08:30:11.027149308 +0000 UTC m=+731.340330148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-online-tls") pod "feast-credit-scoring-86f64d7cd-rbj6g" (UID: "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2") : secret "feast-credit-scoring-online-tls" not found Jul 29 08:30:10.527377 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.527329 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-feast-data\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.529613 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.529596 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-registry-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.555745 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.555713 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt58j\" (UniqueName: \"kubernetes.io/projected/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-kube-api-access-dt58j\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:10.594085 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.594046 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:30:10.745998 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.745970 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:30:10.947961 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:10.947870 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:30:11.031478 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:11.031443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-offline-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:11.031678 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:11.031500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-online-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:11.034029 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:11.033999 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-online-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:11.034178 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:11.034154 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-offline-tls\") pod \"feast-credit-scoring-86f64d7cd-rbj6g\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:11.255048 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:11.255007 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:11.396794 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:11.396770 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g"] Jul 29 08:30:11.399312 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:30:11.399285 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce03067_d5ed_43b5_b7dc_26e2aa9f7bc2.slice/crio-a273a4a75ae8b3fccba72a57d1d5d677ed530fd27850caa05b1f649edd2d62a3 WatchSource:0}: Error finding container a273a4a75ae8b3fccba72a57d1d5d677ed530fd27850caa05b1f649edd2d62a3: Status 404 returned error can't find the container with id a273a4a75ae8b3fccba72a57d1d5d677ed530fd27850caa05b1f649edd2d62a3 Jul 29 08:30:11.625010 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:11.624917 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerStarted","Data":"23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0"} Jul 29 08:30:11.625010 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:11.624998 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerStarted","Data":"a273a4a75ae8b3fccba72a57d1d5d677ed530fd27850caa05b1f649edd2d62a3"} Jul 29 08:30:12.630319 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:12.630285 2582 generic.go:356] "Generic (PLEG): container finished" podID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerID="23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0" exitCode=0 Jul 29 08:30:12.630800 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:12.630385 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerDied","Data":"23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0"} Jul 29 08:30:13.637423 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:13.637393 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerStarted","Data":"58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56"} Jul 29 08:30:19.665461 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:19.665364 2582 generic.go:356] "Generic (PLEG): container finished" podID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerID="58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56" exitCode=0 Jul 29 08:30:19.665461 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:19.665420 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerDied","Data":"58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56"} Jul 29 08:30:20.674320 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:20.674282 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerStarted","Data":"5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9"} Jul 29 08:30:20.674875 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:20.674344 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerStarted","Data":"c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4"} Jul 29 08:30:20.674875 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:20.674363 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerStarted","Data":"1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be"} Jul 29 08:30:20.700315 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:20.700250 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podStartSLOduration=10.700226823 podStartE2EDuration="10.700226823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:30:20.698762223 +0000 UTC m=+741.011943084" watchObservedRunningTime="2026-07-29 08:30:20.700226823 +0000 UTC m=+741.013407685" Jul 29 08:30:21.255427 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:21.255388 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:21.255644 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:21.255456 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:21.255644 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:21.255481 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:23.255538 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:23.255497 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:23.256159 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:23.256137 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:23.256303 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:23.256174 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:23.257291 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:23.257264 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused" Jul 29 08:30:23.257414 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:23.257288 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Jul 29 08:30:23.257414 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:23.257298 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 29 08:30:26.256286 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:26.256241 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Jul 29 08:30:26.256929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:26.256910 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:26.257824 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:26.257799 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:26.261879 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:26.261847 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:26.709636 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:26.709580 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:29.256182 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:29.256124 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Jul 29 08:30:32.256331 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:32.256301 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:32.257158 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:32.257141 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:30:32.581385 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:30:32.581291 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:31:31.498554 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:31.498519 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:31:36.480205 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.480167 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g"] Jul 29 08:31:36.480727 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.480392 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="registry" containerID="cri-o://1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be" gracePeriod=30 Jul 29 08:31:36.480727 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.480451 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="online" containerID="cri-o://c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4" gracePeriod=30 Jul 29 08:31:36.480727 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.480437 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="offline" containerID="cri-o://5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9" gracePeriod=30 Jul 29 08:31:36.505317 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.505285 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6458"] Jul 29 08:31:36.505491 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.505472 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-w6458" podUID="614f25cf-0c52-4cb2-8455-1b12c78b88e5" containerName="postgres" containerID="cri-o://6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19" gracePeriod=30 Jul 29 08:31:36.600367 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.600333 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:31:36.706512 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.706472 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 29 08:31:36.756048 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.756024 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-w6458" Jul 29 08:31:36.878777 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.878736 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb55w\" (UniqueName: \"kubernetes.io/projected/614f25cf-0c52-4cb2-8455-1b12c78b88e5-kube-api-access-mb55w\") pod \"614f25cf-0c52-4cb2-8455-1b12c78b88e5\" (UID: \"614f25cf-0c52-4cb2-8455-1b12c78b88e5\") " Jul 29 08:31:36.878950 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.878858 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/614f25cf-0c52-4cb2-8455-1b12c78b88e5-postgresdata\") pod \"614f25cf-0c52-4cb2-8455-1b12c78b88e5\" (UID: \"614f25cf-0c52-4cb2-8455-1b12c78b88e5\") " Jul 29 08:31:36.881050 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.881016 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614f25cf-0c52-4cb2-8455-1b12c78b88e5-kube-api-access-mb55w" (OuterVolumeSpecName: "kube-api-access-mb55w") pod "614f25cf-0c52-4cb2-8455-1b12c78b88e5" (UID: "614f25cf-0c52-4cb2-8455-1b12c78b88e5"). InnerVolumeSpecName "kube-api-access-mb55w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:31:36.911868 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.911833 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/614f25cf-0c52-4cb2-8455-1b12c78b88e5-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "614f25cf-0c52-4cb2-8455-1b12c78b88e5" (UID: "614f25cf-0c52-4cb2-8455-1b12c78b88e5"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:31:36.980573 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.980537 2582 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/614f25cf-0c52-4cb2-8455-1b12c78b88e5-postgresdata\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:31:36.980573 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:36.980568 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb55w\" (UniqueName: \"kubernetes.io/projected/614f25cf-0c52-4cb2-8455-1b12c78b88e5-kube-api-access-mb55w\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:31:37.014597 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:37.014514 2582 generic.go:356] "Generic (PLEG): container finished" podID="614f25cf-0c52-4cb2-8455-1b12c78b88e5" containerID="6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19" exitCode=0 Jul 29 08:31:37.014597 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:37.014557 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-w6458" Jul 29 08:31:37.014861 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:37.014604 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-w6458" event={"ID":"614f25cf-0c52-4cb2-8455-1b12c78b88e5","Type":"ContainerDied","Data":"6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19"} Jul 29 08:31:37.014861 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:37.014670 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-w6458" event={"ID":"614f25cf-0c52-4cb2-8455-1b12c78b88e5","Type":"ContainerDied","Data":"114c31cd406b9a669898020c957978eeb181300aa0a6cbe8b4865b307e48fd3d"} Jul 29 08:31:37.014861 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:37.014691 2582 scope.go:117] "RemoveContainer" containerID="6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19" Jul 29 08:31:37.023305 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:37.023290 2582 scope.go:117] "RemoveContainer" containerID="6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19" Jul 29 08:31:37.023590 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:31:37.023565 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19\": container with ID starting with 6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19 not found: ID does not exist" containerID="6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19" Jul 29 08:31:37.023822 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:37.023597 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19"} err="failed to get container status \"6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19\": rpc error: code = NotFound desc = could not find container \"6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19\": container with ID starting with 6361efb89305fc9759410e5eb82d6870366fd8221ed542f3f3c8a21c9cb92c19 not found: ID does not exist" Jul 29 08:31:37.036066 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:37.036038 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6458"] Jul 29 08:31:37.039333 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:37.039307 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6458"] Jul 29 08:31:38.021643 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:38.021598 2582 generic.go:356] "Generic (PLEG): container finished" podID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerID="c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4" exitCode=0 Jul 29 08:31:38.022019 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:38.021674 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerDied","Data":"c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4"} Jul 29 08:31:38.250825 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:38.250795 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="614f25cf-0c52-4cb2-8455-1b12c78b88e5" path="/var/lib/kubelet/pods/614f25cf-0c52-4cb2-8455-1b12c78b88e5/volumes" Jul 29 08:31:46.706727 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:46.706687 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 29 08:31:56.706687 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:56.706639 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 29 08:31:56.707169 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:31:56.706769 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:32:06.707301 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:06.707255 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 29 08:32:07.137959 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.137937 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:32:07.139210 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.139186 2582 generic.go:356] "Generic (PLEG): container finished" podID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerID="5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9" exitCode=137 Jul 29 08:32:07.139210 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.139206 2582 generic.go:356] "Generic (PLEG): container finished" podID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerID="1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be" exitCode=137 Jul 29 08:32:07.139419 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.139253 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerDied","Data":"5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9"} Jul 29 08:32:07.139419 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.139324 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerDied","Data":"1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be"} Jul 29 08:32:07.139419 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.139356 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" event={"ID":"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2","Type":"ContainerDied","Data":"a273a4a75ae8b3fccba72a57d1d5d677ed530fd27850caa05b1f649edd2d62a3"} Jul 29 08:32:07.139419 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.139390 2582 scope.go:117] "RemoveContainer" containerID="5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9" Jul 29 08:32:07.147896 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.147878 2582 scope.go:117] "RemoveContainer" containerID="c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4" Jul 29 08:32:07.155950 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.155932 2582 scope.go:117] "RemoveContainer" containerID="1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be" Jul 29 08:32:07.164116 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.164097 2582 scope.go:117] "RemoveContainer" containerID="58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56" Jul 29 08:32:07.171828 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.171808 2582 scope.go:117] "RemoveContainer" containerID="23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0" Jul 29 08:32:07.179291 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.179271 2582 scope.go:117] "RemoveContainer" containerID="5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9" Jul 29 08:32:07.179558 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:32:07.179539 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9\": container with ID starting with 5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9 not found: ID does not exist" containerID="5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9" Jul 29 08:32:07.179685 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.179568 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9"} err="failed to get container status \"5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9\": rpc error: code = NotFound desc = could not find container \"5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9\": container with ID starting with 5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9 not found: ID does not exist" Jul 29 08:32:07.179685 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.179588 2582 scope.go:117] "RemoveContainer" containerID="c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4" Jul 29 08:32:07.179911 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:32:07.179895 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4\": container with ID starting with c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4 not found: ID does not exist" containerID="c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4" Jul 29 08:32:07.179957 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.179918 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4"} err="failed to get container status \"c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4\": rpc error: code = NotFound desc = could not find container \"c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4\": container with ID starting with c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4 not found: ID does not exist" Jul 29 08:32:07.179957 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.179933 2582 scope.go:117] "RemoveContainer" containerID="1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be" Jul 29 08:32:07.180167 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:32:07.180149 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be\": container with ID starting with 1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be not found: ID does not exist" containerID="1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be" Jul 29 08:32:07.180207 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.180183 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be"} err="failed to get container status \"1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be\": rpc error: code = NotFound desc = could not find container \"1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be\": container with ID starting with 1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be not found: ID does not exist" Jul 29 08:32:07.180245 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.180212 2582 scope.go:117] "RemoveContainer" containerID="58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56" Jul 29 08:32:07.180416 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:32:07.180401 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56\": container with ID starting with 58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56 not found: ID does not exist" containerID="58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56" Jul 29 08:32:07.180459 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.180424 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56"} err="failed to get container status \"58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56\": rpc error: code = NotFound desc = could not find container \"58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56\": container with ID starting with 58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56 not found: ID does not exist" Jul 29 08:32:07.180459 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.180436 2582 scope.go:117] "RemoveContainer" containerID="23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0" Jul 29 08:32:07.180649 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:32:07.180609 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0\": container with ID starting with 23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0 not found: ID does not exist" containerID="23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0" Jul 29 08:32:07.180701 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.180654 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0"} err="failed to get container status \"23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0\": rpc error: code = NotFound desc = could not find container \"23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0\": container with ID starting with 23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0 not found: ID does not exist" Jul 29 08:32:07.180701 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.180670 2582 scope.go:117] "RemoveContainer" containerID="5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9" Jul 29 08:32:07.180921 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.180901 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9"} err="failed to get container status \"5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9\": rpc error: code = NotFound desc = could not find container \"5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9\": container with ID starting with 5562315fa6343b454a6897f90745042ec31546a6c39c811e7584b350fa8308b9 not found: ID does not exist" Jul 29 08:32:07.180987 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.180924 2582 scope.go:117] "RemoveContainer" containerID="c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4" Jul 29 08:32:07.181178 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.181162 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4"} err="failed to get container status \"c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4\": rpc error: code = NotFound desc = could not find container \"c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4\": container with ID starting with c2e1fcb56901a90fc829e8605736bfc47a27d417caa4757219a621c79dd7f7e4 not found: ID does not exist" Jul 29 08:32:07.181273 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.181194 2582 scope.go:117] "RemoveContainer" containerID="1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be" Jul 29 08:32:07.181457 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.181438 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be"} err="failed to get container status \"1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be\": rpc error: code = NotFound desc = could not find container \"1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be\": container with ID starting with 1b914f6bfa54ad0a93d5e7e2f670a6cb58c19e45dbabecae028880c9184820be not found: ID does not exist" Jul 29 08:32:07.181507 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.181457 2582 scope.go:117] "RemoveContainer" containerID="58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56" Jul 29 08:32:07.181709 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.181690 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56"} err="failed to get container status \"58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56\": rpc error: code = NotFound desc = could not find container \"58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56\": container with ID starting with 58df66eca8db6a37110e7180e836ac3347d0c1458269b89d82e0a5381a8dcc56 not found: ID does not exist" Jul 29 08:32:07.181764 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.181711 2582 scope.go:117] "RemoveContainer" containerID="23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0" Jul 29 08:32:07.181925 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.181905 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0"} err="failed to get container status \"23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0\": rpc error: code = NotFound desc = could not find container \"23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0\": container with ID starting with 23945efec837c28a41e1296a55bbd23da2ee0901d4e8cd0ea7b16d2053f3ecf0 not found: ID does not exist" Jul 29 08:32:07.245696 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.245651 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-registry-tls\") pod \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " Jul 29 08:32:07.245896 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.245707 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-offline-tls\") pod \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " Jul 29 08:32:07.245896 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.245749 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-online-tls\") pod \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " Jul 29 08:32:07.245896 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.245776 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt58j\" (UniqueName: \"kubernetes.io/projected/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-kube-api-access-dt58j\") pod \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " Jul 29 08:32:07.245896 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.245803 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-feast-data\") pod \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\" (UID: \"fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2\") " Jul 29 08:32:07.248114 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.248081 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" (UID: "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:32:07.248114 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.248095 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-kube-api-access-dt58j" (OuterVolumeSpecName: "kube-api-access-dt58j") pod "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" (UID: "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2"). InnerVolumeSpecName "kube-api-access-dt58j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:32:07.248359 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.248340 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" (UID: "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:32:07.248414 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.248357 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-online-tls" (OuterVolumeSpecName: "online-tls") pod "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" (UID: "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:32:07.253718 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.253687 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-feast-data" (OuterVolumeSpecName: "feast-data") pod "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" (UID: "fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:32:07.346975 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.346936 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:32:07.346975 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.346967 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:32:07.346975 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.346975 2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-offline-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:32:07.346975 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.346984 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:32:07.347237 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:07.346992 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt58j\" (UniqueName: \"kubernetes.io/projected/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2-kube-api-access-dt58j\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:32:08.144021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:08.143988 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g" Jul 29 08:32:08.184591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:08.184558 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g"] Jul 29 08:32:08.196735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:08.196710 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-86f64d7cd-rbj6g"] Jul 29 08:32:08.251420 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:08.251390 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" path="/var/lib/kubelet/pods/fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2/volumes" Jul 29 08:32:13.704793 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:13.704757 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:32:13.800538 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:13.800508 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:32:14.959804 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:14.959768 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:32:15.356924 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:32:15.356835 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:17.560669 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.560633 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:17.696032 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.695999 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-9lpff"] Jul 29 08:34:17.696323 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696311 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="feast-apply" Jul 29 08:34:17.696382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696325 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="feast-apply" Jul 29 08:34:17.696382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696343 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="feast-init" Jul 29 08:34:17.696382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696348 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="feast-init" Jul 29 08:34:17.696382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696354 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="online" Jul 29 08:34:17.696382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696359 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="online" Jul 29 08:34:17.696382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696370 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614f25cf-0c52-4cb2-8455-1b12c78b88e5" containerName="postgres" Jul 29 08:34:17.696382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696375 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="614f25cf-0c52-4cb2-8455-1b12c78b88e5" containerName="postgres" Jul 29 08:34:17.696382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696380 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="offline" Jul 29 08:34:17.696382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696386 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="offline" Jul 29 08:34:17.696716 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696399 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="registry" Jul 29 08:34:17.696716 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696404 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="registry" Jul 29 08:34:17.696716 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696450 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="online" Jul 29 08:34:17.696716 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696457 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="offline" Jul 29 08:34:17.696716 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696464 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce03067-d5ed-43b5-b7dc-26e2aa9f7bc2" containerName="registry" Jul 29 08:34:17.696716 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.696471 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="614f25cf-0c52-4cb2-8455-1b12c78b88e5" containerName="postgres" Jul 29 08:34:17.699090 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.699073 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" Jul 29 08:34:17.701392 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.701358 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jul 29 08:34:17.701503 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.701368 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jul 29 08:34:17.701970 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.701954 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-fj9wn\"" Jul 29 08:34:17.715514 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.715492 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-9lpff"] Jul 29 08:34:17.856326 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.856237 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgdvt\" (UniqueName: \"kubernetes.io/projected/8b373889-0841-4fac-a1ef-41a9545e76e4-kube-api-access-wgdvt\") pod \"redis-69bbd798b6-9lpff\" (UID: \"8b373889-0841-4fac-a1ef-41a9545e76e4\") " pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" Jul 29 08:34:17.957368 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.957332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdvt\" (UniqueName: \"kubernetes.io/projected/8b373889-0841-4fac-a1ef-41a9545e76e4-kube-api-access-wgdvt\") pod \"redis-69bbd798b6-9lpff\" (UID: \"8b373889-0841-4fac-a1ef-41a9545e76e4\") " pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" Jul 29 08:34:17.966815 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:17.966785 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdvt\" (UniqueName: \"kubernetes.io/projected/8b373889-0841-4fac-a1ef-41a9545e76e4-kube-api-access-wgdvt\") pod \"redis-69bbd798b6-9lpff\" (UID: \"8b373889-0841-4fac-a1ef-41a9545e76e4\") " pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" Jul 29 08:34:18.016159 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:18.016132 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" Jul 29 08:34:18.162104 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:18.162029 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-9lpff"] Jul 29 08:34:18.165364 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:34:18.165325 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b373889_0841_4fac_a1ef_41a9545e76e4.slice/crio-69426aa530828130b93c9eee79f59d7c3f2f59aadc5f4df8ee9bbed47ff1ad64 WatchSource:0}: Error finding container 69426aa530828130b93c9eee79f59d7c3f2f59aadc5f4df8ee9bbed47ff1ad64: Status 404 returned error can't find the container with id 69426aa530828130b93c9eee79f59d7c3f2f59aadc5f4df8ee9bbed47ff1ad64 Jul 29 08:34:18.658677 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:18.658644 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" event={"ID":"8b373889-0841-4fac-a1ef-41a9545e76e4","Type":"ContainerStarted","Data":"69426aa530828130b93c9eee79f59d7c3f2f59aadc5f4df8ee9bbed47ff1ad64"} Jul 29 08:34:18.986949 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:18.986916 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:20.667532 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:20.667455 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" event={"ID":"8b373889-0841-4fac-a1ef-41a9545e76e4","Type":"ContainerStarted","Data":"8c9baae00cc3b7a33b4d03e2b85e25a356df39a3ef98c1f57c88a5f8a6be4237"} Jul 29 08:34:23.137876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.137822 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" podStartSLOduration=4.634028161 podStartE2EDuration="6.137803713s" totalImagesPullingTime="1.503775552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:34:20.683037026 +0000 UTC m=+980.996217885" watchObservedRunningTime="2026-07-29 08:34:23.137803713 +0000 UTC m=+983.450984590" Jul 29 08:34:23.138790 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.138768 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5r2vs"] Jul 29 08:34:23.142613 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.142592 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" Jul 29 08:34:23.150343 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.149863 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jul 29 08:34:23.150343 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.149879 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5r2vs"] Jul 29 08:34:23.172528 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.172479 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:23.304086 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.304041 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/fc436136-fa7f-4931-ac1d-c51f83824177-postgresdata\") pod \"postgres-864846d4f5-5r2vs\" (UID: \"fc436136-fa7f-4931-ac1d-c51f83824177\") " pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" Jul 29 08:34:23.304268 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.304095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2rgn\" (UniqueName: \"kubernetes.io/projected/fc436136-fa7f-4931-ac1d-c51f83824177-kube-api-access-r2rgn\") pod \"postgres-864846d4f5-5r2vs\" (UID: \"fc436136-fa7f-4931-ac1d-c51f83824177\") " pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" Jul 29 08:34:23.405263 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.405169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/fc436136-fa7f-4931-ac1d-c51f83824177-postgresdata\") pod \"postgres-864846d4f5-5r2vs\" (UID: \"fc436136-fa7f-4931-ac1d-c51f83824177\") " pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" Jul 29 08:34:23.405263 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.405213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2rgn\" (UniqueName: \"kubernetes.io/projected/fc436136-fa7f-4931-ac1d-c51f83824177-kube-api-access-r2rgn\") pod \"postgres-864846d4f5-5r2vs\" (UID: \"fc436136-fa7f-4931-ac1d-c51f83824177\") " pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" Jul 29 08:34:23.405691 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.405672 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/fc436136-fa7f-4931-ac1d-c51f83824177-postgresdata\") pod \"postgres-864846d4f5-5r2vs\" (UID: \"fc436136-fa7f-4931-ac1d-c51f83824177\") " pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" Jul 29 08:34:23.414010 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.413985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2rgn\" (UniqueName: \"kubernetes.io/projected/fc436136-fa7f-4931-ac1d-c51f83824177-kube-api-access-r2rgn\") pod \"postgres-864846d4f5-5r2vs\" (UID: \"fc436136-fa7f-4931-ac1d-c51f83824177\") " pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" Jul 29 08:34:23.453414 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.453385 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" Jul 29 08:34:23.587751 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.587589 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5r2vs"] Jul 29 08:34:23.590382 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:34:23.590355 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc436136_fa7f_4931_ac1d_c51f83824177.slice/crio-688bee981ad2d2671b94a2a3660f38e0c83c2b186174a004b817b37ff4a4ecff WatchSource:0}: Error finding container 688bee981ad2d2671b94a2a3660f38e0c83c2b186174a004b817b37ff4a4ecff: Status 404 returned error can't find the container with id 688bee981ad2d2671b94a2a3660f38e0c83c2b186174a004b817b37ff4a4ecff Jul 29 08:34:23.679890 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.679856 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" event={"ID":"fc436136-fa7f-4931-ac1d-c51f83824177","Type":"ContainerStarted","Data":"e2deb87263f8fb75cc77fdee8f44a472f7c919b5c429aa03b5a4388805ba1c4f"} Jul 29 08:34:23.680045 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.679903 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" event={"ID":"fc436136-fa7f-4931-ac1d-c51f83824177","Type":"ContainerStarted","Data":"688bee981ad2d2671b94a2a3660f38e0c83c2b186174a004b817b37ff4a4ecff"} Jul 29 08:34:23.694939 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:23.694893 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" podStartSLOduration=0.694877345 podStartE2EDuration="694.877345ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:34:23.693931981 +0000 UTC m=+984.007112843" watchObservedRunningTime="2026-07-29 08:34:23.694877345 +0000 UTC m=+984.008058235" Jul 29 08:34:28.646890 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.646848 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:28.735865 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.735821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:28.739698 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.739672 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg"] Jul 29 08:34:28.743155 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.743138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.745553 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.745535 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jul 29 08:34:28.745692 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.745555 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jul 29 08:34:28.745692 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.745574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-registry-tls\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.745692 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.745560 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jul 29 08:34:28.745692 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.745645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-online-tls\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.745692 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.745565 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-2rf7q\"" Jul 29 08:34:28.745965 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.745754 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsm76\" (UniqueName: \"kubernetes.io/projected/be0fc037-bbcb-401a-87a2-489c5bee7e31-kube-api-access-jsm76\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.745965 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.745811 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be0fc037-bbcb-401a-87a2-489c5bee7e31-feast-data\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.756491 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.756462 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg"] Jul 29 08:34:28.804766 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.804738 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:28.846162 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.846124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-online-tls\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.846347 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.846212 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsm76\" (UniqueName: \"kubernetes.io/projected/be0fc037-bbcb-401a-87a2-489c5bee7e31-kube-api-access-jsm76\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.846347 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.846240 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be0fc037-bbcb-401a-87a2-489c5bee7e31-feast-data\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.846347 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.846267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-registry-tls\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.846704 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.846674 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be0fc037-bbcb-401a-87a2-489c5bee7e31-feast-data\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.848725 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.848698 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-online-tls\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.848839 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.848790 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-registry-tls\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.857702 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.857680 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsm76\" (UniqueName: \"kubernetes.io/projected/be0fc037-bbcb-401a-87a2-489c5bee7e31-kube-api-access-jsm76\") pod \"feast-credit-scoring-5859ff5d89-rhbfg\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:28.878123 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:28.878096 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:29.057970 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:29.057938 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:29.195748 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:29.195715 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg"] Jul 29 08:34:29.198684 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:34:29.198656 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0fc037_bbcb_401a_87a2_489c5bee7e31.slice/crio-6de1b3a56a0d5aacfea4c1dde0d5d61a34041f2294fe4022f0fea819bc102eee WatchSource:0}: Error finding container 6de1b3a56a0d5aacfea4c1dde0d5d61a34041f2294fe4022f0fea819bc102eee: Status 404 returned error can't find the container with id 6de1b3a56a0d5aacfea4c1dde0d5d61a34041f2294fe4022f0fea819bc102eee Jul 29 08:34:29.706987 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:29.706952 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" event={"ID":"be0fc037-bbcb-401a-87a2-489c5bee7e31","Type":"ContainerStarted","Data":"0c88a9a386e708a7406819ed5a435da8a75dd93aa35841c78ecfef304a90760e"} Jul 29 08:34:29.707367 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:29.707010 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" event={"ID":"be0fc037-bbcb-401a-87a2-489c5bee7e31","Type":"ContainerStarted","Data":"6de1b3a56a0d5aacfea4c1dde0d5d61a34041f2294fe4022f0fea819bc102eee"} Jul 29 08:34:30.712697 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:30.712662 2582 generic.go:356] "Generic (PLEG): container finished" podID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerID="0c88a9a386e708a7406819ed5a435da8a75dd93aa35841c78ecfef304a90760e" exitCode=0 Jul 29 08:34:30.713099 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:30.712756 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" event={"ID":"be0fc037-bbcb-401a-87a2-489c5bee7e31","Type":"ContainerDied","Data":"0c88a9a386e708a7406819ed5a435da8a75dd93aa35841c78ecfef304a90760e"} Jul 29 08:34:31.719105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:31.719070 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" event={"ID":"be0fc037-bbcb-401a-87a2-489c5bee7e31","Type":"ContainerStarted","Data":"359e0473d6d87da2afa75421be05d02883afe67f4d5fffd3c8558151e082292f"} Jul 29 08:34:37.747875 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:37.747836 2582 generic.go:356] "Generic (PLEG): container finished" podID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerID="359e0473d6d87da2afa75421be05d02883afe67f4d5fffd3c8558151e082292f" exitCode=0 Jul 29 08:34:37.748384 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:37.747888 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" event={"ID":"be0fc037-bbcb-401a-87a2-489c5bee7e31","Type":"ContainerDied","Data":"359e0473d6d87da2afa75421be05d02883afe67f4d5fffd3c8558151e082292f"} Jul 29 08:34:38.756104 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:38.756051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" event={"ID":"be0fc037-bbcb-401a-87a2-489c5bee7e31","Type":"ContainerStarted","Data":"d91a6b818f02f1dbfd8d21fa20f5b0daf62361c8b77a8fed9206e10408daddfc"} Jul 29 08:34:38.756483 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:38.756144 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" event={"ID":"be0fc037-bbcb-401a-87a2-489c5bee7e31","Type":"ContainerStarted","Data":"71185af92b5b654cb15b273a7a5292dcafd1d3a822efc03bd8db92304c56045f"} Jul 29 08:34:38.786037 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:38.785987 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" podStartSLOduration=10.785971889 podStartE2EDuration="10.785971889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:34:38.785723358 +0000 UTC m=+999.098904234" watchObservedRunningTime="2026-07-29 08:34:38.785971889 +0000 UTC m=+999.099152748" Jul 29 08:34:39.058560 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:39.058474 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:39.058723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:39.058568 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:41.058820 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:41.058784 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:41.059225 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:41.058854 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:41.060406 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:41.060381 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused" Jul 29 08:34:41.060502 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:41.060423 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 29 08:34:44.058747 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:44.058690 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused" Jul 29 08:34:44.064176 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:44.064152 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:44.067460 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:44.067435 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:47.059138 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:47.059092 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused" Jul 29 08:34:50.059563 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:50.059533 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:50.060316 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:50.060298 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:34:50.427187 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:50.427059 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:54.570392 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.570357 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:54.672904 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.672870 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:54.676561 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.676532 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f"] Jul 29 08:34:54.680844 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.680829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:54.684757 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.684738 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jul 29 08:34:54.684874 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.684767 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-hgwth\"" Jul 29 08:34:54.684996 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.684982 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jul 29 08:34:54.685042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.684987 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jul 29 08:34:54.696011 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.695991 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f"] Jul 29 08:34:54.780349 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.780311 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbwfc\" (UniqueName: \"kubernetes.io/projected/26ec9b5d-9658-47a8-857f-8f210bc83d13-kube-api-access-sbwfc\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:54.780349 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.780347 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-online-tls\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:54.780553 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.780378 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/26ec9b5d-9658-47a8-857f-8f210bc83d13-feast-data\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:54.780553 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.780444 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-registry-tls\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:54.881477 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.881394 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/26ec9b5d-9658-47a8-857f-8f210bc83d13-feast-data\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:54.881477 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.881436 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-registry-tls\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:54.881714 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.881492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbwfc\" (UniqueName: \"kubernetes.io/projected/26ec9b5d-9658-47a8-857f-8f210bc83d13-kube-api-access-sbwfc\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:54.881714 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.881512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-online-tls\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:54.881714 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:34:54.881605 2582 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Jul 29 08:34:54.881714 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:34:54.881609 2582 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Jul 29 08:34:54.881714 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:34:54.881698 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-registry-tls podName:26ec9b5d-9658-47a8-857f-8f210bc83d13 nodeName:}" failed. No retries permitted until 2026-07-29 08:34:55.381679895 +0000 UTC m=+1015.694860733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-registry-tls") pod "feast-driver-ranking-5976578f46-k5h7f" (UID: "26ec9b5d-9658-47a8-857f-8f210bc83d13") : secret "feast-driver-ranking-registry-tls" not found Jul 29 08:34:54.881885 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:34:54.881723 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-online-tls podName:26ec9b5d-9658-47a8-857f-8f210bc83d13 nodeName:}" failed. No retries permitted until 2026-07-29 08:34:55.381712136 +0000 UTC m=+1015.694892976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-online-tls") pod "feast-driver-ranking-5976578f46-k5h7f" (UID: "26ec9b5d-9658-47a8-857f-8f210bc83d13") : secret "feast-driver-ranking-online-tls" not found Jul 29 08:34:54.881885 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.881796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/26ec9b5d-9658-47a8-857f-8f210bc83d13-feast-data\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:54.893368 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:54.893336 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbwfc\" (UniqueName: \"kubernetes.io/projected/26ec9b5d-9658-47a8-857f-8f210bc83d13-kube-api-access-sbwfc\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:55.056238 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:55.056205 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:55.190494 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:55.190412 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:34:55.384660 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:55.384608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-registry-tls\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:55.384851 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:55.384765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-online-tls\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:55.387292 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:55.387270 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-registry-tls\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:55.387405 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:55.387325 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-online-tls\") pod \"feast-driver-ranking-5976578f46-k5h7f\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:55.589766 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:55.589733 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:34:55.735381 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:55.735350 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f"] Jul 29 08:34:55.738509 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:34:55.738484 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ec9b5d_9658_47a8_857f_8f210bc83d13.slice/crio-44d33fce0e52cf37820f7a4e797ed34f8b8e8c3b2822bc3870194dabee87994f WatchSource:0}: Error finding container 44d33fce0e52cf37820f7a4e797ed34f8b8e8c3b2822bc3870194dabee87994f: Status 404 returned error can't find the container with id 44d33fce0e52cf37820f7a4e797ed34f8b8e8c3b2822bc3870194dabee87994f Jul 29 08:34:55.836360 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:55.836323 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" event={"ID":"26ec9b5d-9658-47a8-857f-8f210bc83d13","Type":"ContainerStarted","Data":"51683bc89281a31b6df4579bb664429c536d21cedaeaa823459b0e857650c53f"} Jul 29 08:34:55.836520 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:34:55.836383 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" event={"ID":"26ec9b5d-9658-47a8-857f-8f210bc83d13","Type":"ContainerStarted","Data":"44d33fce0e52cf37820f7a4e797ed34f8b8e8c3b2822bc3870194dabee87994f"} Jul 29 08:35:00.862193 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:00.862105 2582 generic.go:356] "Generic (PLEG): container finished" podID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerID="51683bc89281a31b6df4579bb664429c536d21cedaeaa823459b0e857650c53f" exitCode=0 Jul 29 08:35:00.862193 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:00.862159 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" event={"ID":"26ec9b5d-9658-47a8-857f-8f210bc83d13","Type":"ContainerDied","Data":"51683bc89281a31b6df4579bb664429c536d21cedaeaa823459b0e857650c53f"} Jul 29 08:35:01.869749 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:01.869712 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" event={"ID":"26ec9b5d-9658-47a8-857f-8f210bc83d13","Type":"ContainerStarted","Data":"fec26e3526a7ff4e3bcb41442ab95b326e6fe762ec9cb14b65777b1e58bcce52"} Jul 29 08:35:04.960702 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:04.960661 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:35:05.896367 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:05.896334 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:35:07.898501 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:07.898409 2582 generic.go:356] "Generic (PLEG): container finished" podID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerID="fec26e3526a7ff4e3bcb41442ab95b326e6fe762ec9cb14b65777b1e58bcce52" exitCode=0 Jul 29 08:35:07.898501 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:07.898449 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" event={"ID":"26ec9b5d-9658-47a8-857f-8f210bc83d13","Type":"ContainerDied","Data":"fec26e3526a7ff4e3bcb41442ab95b326e6fe762ec9cb14b65777b1e58bcce52"} Jul 29 08:35:07.902130 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:07.902109 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jul 29 08:35:08.906307 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:08.906277 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" event={"ID":"26ec9b5d-9658-47a8-857f-8f210bc83d13","Type":"ContainerStarted","Data":"faae5951a79b70170e377ad3f30b481e3cf4067f363a009ead5d2ed8a18016a4"} Jul 29 08:35:08.906721 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:08.906330 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" event={"ID":"26ec9b5d-9658-47a8-857f-8f210bc83d13","Type":"ContainerStarted","Data":"6e1339bcee8ac31d7d600ae682c816af383e3f8602ce5f813129da0987bcf778"} Jul 29 08:35:08.934163 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:08.934102 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podStartSLOduration=14.934083567 podStartE2EDuration="14.934083567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:35:08.931546332 +0000 UTC m=+1029.244727217" watchObservedRunningTime="2026-07-29 08:35:08.934083567 +0000 UTC m=+1029.247264430" Jul 29 08:35:10.590184 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:10.590144 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:35:10.590587 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:10.590261 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:35:10.591903 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:10.591877 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Jul 29 08:35:10.591903 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:10.591890 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused" Jul 29 08:35:13.590323 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:13.590280 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused" Jul 29 08:35:13.590918 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:13.590363 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Jul 29 08:35:15.590127 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:15.590094 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:35:15.590657 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:15.590166 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:35:16.591087 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:16.591037 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.36:6571: connect: connection refused" Jul 29 08:35:16.596918 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:16.596891 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:35:16.600337 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:16.600306 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:35:19.591370 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:19.591333 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:35:19.592313 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:35:19.592294 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:36:32.237897 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.237859 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:32.786007 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.785972 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-648d8678b4-dg7bw"] Jul 29 08:36:32.789548 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.789527 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:32.807074 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.807050 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648d8678b4-dg7bw"] Jul 29 08:36:32.850056 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.850021 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:32.916201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.916166 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:32.963834 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.963801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vnh\" (UniqueName: \"kubernetes.io/projected/4447f42f-6753-4a4e-8f01-5f57a64b5c75-kube-api-access-96vnh\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:32.964016 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.963847 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4447f42f-6753-4a4e-8f01-5f57a64b5c75-console-oauth-config\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:32.964016 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.963872 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4447f42f-6753-4a4e-8f01-5f57a64b5c75-console-serving-cert\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:32.964016 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.963896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-oauth-serving-cert\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:32.964016 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.963940 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-service-ca\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:32.964182 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.964050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-console-config\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:32.964182 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:32.964078 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-trusted-ca-bundle\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.065570 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.065457 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-console-config\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.065570 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.065508 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-trusted-ca-bundle\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.065854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.065673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vnh\" (UniqueName: \"kubernetes.io/projected/4447f42f-6753-4a4e-8f01-5f57a64b5c75-kube-api-access-96vnh\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.065854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.065731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4447f42f-6753-4a4e-8f01-5f57a64b5c75-console-oauth-config\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.065854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.065757 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4447f42f-6753-4a4e-8f01-5f57a64b5c75-console-serving-cert\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.065854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.065770 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-oauth-serving-cert\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.065854 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.065789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-service-ca\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.066344 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.066310 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-console-config\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.066478 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.066448 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-oauth-serving-cert\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.066538 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.066478 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-service-ca\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.066538 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.066485 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4447f42f-6753-4a4e-8f01-5f57a64b5c75-trusted-ca-bundle\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.068310 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.068284 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4447f42f-6753-4a4e-8f01-5f57a64b5c75-console-oauth-config\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.068401 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.068383 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4447f42f-6753-4a4e-8f01-5f57a64b5c75-console-serving-cert\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.074614 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.074594 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vnh\" (UniqueName: \"kubernetes.io/projected/4447f42f-6753-4a4e-8f01-5f57a64b5c75-kube-api-access-96vnh\") pod \"console-648d8678b4-dg7bw\" (UID: \"4447f42f-6753-4a4e-8f01-5f57a64b5c75\") " pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.098732 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.098698 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:33.251118 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.251095 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648d8678b4-dg7bw"] Jul 29 08:36:33.253198 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:36:33.253162 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4447f42f_6753_4a4e_8f01_5f57a64b5c75.slice/crio-4f2431077fe2da0d1578b3119b68cfa00d0dbcfeef67cbe30ce163f240530ce3 WatchSource:0}: Error finding container 4f2431077fe2da0d1578b3119b68cfa00d0dbcfeef67cbe30ce163f240530ce3: Status 404 returned error can't find the container with id 4f2431077fe2da0d1578b3119b68cfa00d0dbcfeef67cbe30ce163f240530ce3 Jul 29 08:36:33.343195 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.343107 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648d8678b4-dg7bw" event={"ID":"4447f42f-6753-4a4e-8f01-5f57a64b5c75","Type":"ContainerStarted","Data":"2be6810bd9fbf5e53f895f32ccc24d1e558959383d6138328d8d7f956d99348e"} Jul 29 08:36:33.343195 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.343159 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648d8678b4-dg7bw" event={"ID":"4447f42f-6753-4a4e-8f01-5f57a64b5c75","Type":"ContainerStarted","Data":"4f2431077fe2da0d1578b3119b68cfa00d0dbcfeef67cbe30ce163f240530ce3"} Jul 29 08:36:33.363494 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:33.363430 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-648d8678b4-dg7bw" podStartSLOduration=1.363408292 podStartE2EDuration="1.363408292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:36:33.362173268 +0000 UTC m=+1113.675354130" watchObservedRunningTime="2026-07-29 08:36:33.363408292 +0000 UTC m=+1113.676589152" Jul 29 08:36:34.253847 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:34.253809 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:34.691112 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:34.691024 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6"] Jul 29 08:36:34.691258 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:34.691234 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" podUID="3d950279-670c-4203-94ae-46c922d2b1f6" containerName="manager" containerID="cri-o://92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba" gracePeriod=10 Jul 29 08:36:34.700733 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:34.700706 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:34.959511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:34.959485 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" Jul 29 08:36:35.005342 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.005283 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:35.055379 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.055347 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg"] Jul 29 08:36:35.055592 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.055561 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="registry" containerID="cri-o://71185af92b5b654cb15b273a7a5292dcafd1d3a822efc03bd8db92304c56045f" gracePeriod=30 Jul 29 08:36:35.055761 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.055613 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="online" containerID="cri-o://d91a6b818f02f1dbfd8d21fa20f5b0daf62361c8b77a8fed9206e10408daddfc" gracePeriod=30 Jul 29 08:36:35.081852 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.081812 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f"] Jul 29 08:36:35.082077 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.082050 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="registry" containerID="cri-o://6e1339bcee8ac31d7d600ae682c816af383e3f8602ce5f813129da0987bcf778" gracePeriod=30 Jul 29 08:36:35.082143 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.082090 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="online" containerID="cri-o://faae5951a79b70170e377ad3f30b481e3cf4067f363a009ead5d2ed8a18016a4" gracePeriod=30 Jul 29 08:36:35.085447 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.085424 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmfz6\" (UniqueName: \"kubernetes.io/projected/3d950279-670c-4203-94ae-46c922d2b1f6-kube-api-access-rmfz6\") pod \"3d950279-670c-4203-94ae-46c922d2b1f6\" (UID: \"3d950279-670c-4203-94ae-46c922d2b1f6\") " Jul 29 08:36:35.087553 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.087532 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d950279-670c-4203-94ae-46c922d2b1f6-kube-api-access-rmfz6" (OuterVolumeSpecName: "kube-api-access-rmfz6") pod "3d950279-670c-4203-94ae-46c922d2b1f6" (UID: "3d950279-670c-4203-94ae-46c922d2b1f6"). InnerVolumeSpecName "kube-api-access-rmfz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:36:35.110128 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.110096 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:35.186337 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.186301 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmfz6\" (UniqueName: \"kubernetes.io/projected/3d950279-670c-4203-94ae-46c922d2b1f6-kube-api-access-rmfz6\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:36:35.353545 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.353502 2582 generic.go:356] "Generic (PLEG): container finished" podID="3d950279-670c-4203-94ae-46c922d2b1f6" containerID="92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba" exitCode=0 Jul 29 08:36:35.354029 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.353546 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" event={"ID":"3d950279-670c-4203-94ae-46c922d2b1f6","Type":"ContainerDied","Data":"92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba"} Jul 29 08:36:35.354029 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.353553 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" Jul 29 08:36:35.354029 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.353597 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6" event={"ID":"3d950279-670c-4203-94ae-46c922d2b1f6","Type":"ContainerDied","Data":"02505fd70d620b3738f60380f0344a5592a4ce4813aed34cbd8ffa173eb4de26"} Jul 29 08:36:35.354029 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.353642 2582 scope.go:117] "RemoveContainer" containerID="92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba" Jul 29 08:36:35.363396 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.363375 2582 scope.go:117] "RemoveContainer" containerID="92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba" Jul 29 08:36:35.363713 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:36:35.363693 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba\": container with ID starting with 92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba not found: ID does not exist" containerID="92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba" Jul 29 08:36:35.363772 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.363722 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba"} err="failed to get container status \"92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba\": rpc error: code = NotFound desc = could not find container \"92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba\": container with ID starting with 92a272985ff0571502bf4368ce2f49f5145a297265c654bad2f4a4eb0e8f70ba not found: ID does not exist" Jul 29 08:36:35.386857 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.386828 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6"] Jul 29 08:36:35.391220 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:35.391182 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f54bd498c-zfxg6"] Jul 29 08:36:36.252177 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:36.252142 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d950279-670c-4203-94ae-46c922d2b1f6" path="/var/lib/kubelet/pods/3d950279-670c-4203-94ae-46c922d2b1f6/volumes" Jul 29 08:36:36.360426 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:36.360396 2582 generic.go:356] "Generic (PLEG): container finished" podID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerID="d91a6b818f02f1dbfd8d21fa20f5b0daf62361c8b77a8fed9206e10408daddfc" exitCode=0 Jul 29 08:36:36.360838 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:36.360467 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" event={"ID":"be0fc037-bbcb-401a-87a2-489c5bee7e31","Type":"ContainerDied","Data":"d91a6b818f02f1dbfd8d21fa20f5b0daf62361c8b77a8fed9206e10408daddfc"} Jul 29 08:36:36.362756 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:36.362736 2582 generic.go:356] "Generic (PLEG): container finished" podID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerID="faae5951a79b70170e377ad3f30b481e3cf4067f363a009ead5d2ed8a18016a4" exitCode=0 Jul 29 08:36:36.362826 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:36.362807 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" event={"ID":"26ec9b5d-9658-47a8-857f-8f210bc83d13","Type":"ContainerDied","Data":"faae5951a79b70170e377ad3f30b481e3cf4067f363a009ead5d2ed8a18016a4"} Jul 29 08:36:36.598195 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:36.598097 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Jul 29 08:36:37.181752 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.181713 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5r2vs"] Jul 29 08:36:37.181932 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.181894 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" podUID="fc436136-fa7f-4931-ac1d-c51f83824177" containerName="postgres" containerID="cri-o://e2deb87263f8fb75cc77fdee8f44a472f7c919b5c429aa03b5a4388805ba1c4f" gracePeriod=30 Jul 29 08:36:37.189134 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.189100 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-9lpff"] Jul 29 08:36:37.189339 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.189306 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" podUID="8b373889-0841-4fac-a1ef-41a9545e76e4" containerName="redis" containerID="cri-o://8c9baae00cc3b7a33b4d03e2b85e25a356df39a3ef98c1f57c88a5f8a6be4237" gracePeriod=30 Jul 29 08:36:37.266019 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.265989 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:37.370377 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.370347 2582 generic.go:356] "Generic (PLEG): container finished" podID="fc436136-fa7f-4931-ac1d-c51f83824177" containerID="e2deb87263f8fb75cc77fdee8f44a472f7c919b5c429aa03b5a4388805ba1c4f" exitCode=0 Jul 29 08:36:37.370843 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.370424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" event={"ID":"fc436136-fa7f-4931-ac1d-c51f83824177","Type":"ContainerDied","Data":"e2deb87263f8fb75cc77fdee8f44a472f7c919b5c429aa03b5a4388805ba1c4f"} Jul 29 08:36:37.372030 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.372008 2582 generic.go:356] "Generic (PLEG): container finished" podID="8b373889-0841-4fac-a1ef-41a9545e76e4" containerID="8c9baae00cc3b7a33b4d03e2b85e25a356df39a3ef98c1f57c88a5f8a6be4237" exitCode=0 Jul 29 08:36:37.372160 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.372062 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" event={"ID":"8b373889-0841-4fac-a1ef-41a9545e76e4","Type":"ContainerDied","Data":"8c9baae00cc3b7a33b4d03e2b85e25a356df39a3ef98c1f57c88a5f8a6be4237"} Jul 29 08:36:37.464462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.464435 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" Jul 29 08:36:37.467302 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.467282 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" Jul 29 08:36:37.606364 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.606340 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2rgn\" (UniqueName: \"kubernetes.io/projected/fc436136-fa7f-4931-ac1d-c51f83824177-kube-api-access-r2rgn\") pod \"fc436136-fa7f-4931-ac1d-c51f83824177\" (UID: \"fc436136-fa7f-4931-ac1d-c51f83824177\") " Jul 29 08:36:37.606533 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.606374 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgdvt\" (UniqueName: \"kubernetes.io/projected/8b373889-0841-4fac-a1ef-41a9545e76e4-kube-api-access-wgdvt\") pod \"8b373889-0841-4fac-a1ef-41a9545e76e4\" (UID: \"8b373889-0841-4fac-a1ef-41a9545e76e4\") " Jul 29 08:36:37.606533 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.606422 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/fc436136-fa7f-4931-ac1d-c51f83824177-postgresdata\") pod \"fc436136-fa7f-4931-ac1d-c51f83824177\" (UID: \"fc436136-fa7f-4931-ac1d-c51f83824177\") " Jul 29 08:36:37.608723 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.608688 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc436136-fa7f-4931-ac1d-c51f83824177-kube-api-access-r2rgn" (OuterVolumeSpecName: "kube-api-access-r2rgn") pod "fc436136-fa7f-4931-ac1d-c51f83824177" (UID: "fc436136-fa7f-4931-ac1d-c51f83824177"). InnerVolumeSpecName "kube-api-access-r2rgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:36:37.608839 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.608726 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b373889-0841-4fac-a1ef-41a9545e76e4-kube-api-access-wgdvt" (OuterVolumeSpecName: "kube-api-access-wgdvt") pod "8b373889-0841-4fac-a1ef-41a9545e76e4" (UID: "8b373889-0841-4fac-a1ef-41a9545e76e4"). InnerVolumeSpecName "kube-api-access-wgdvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:36:37.636286 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.636253 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc436136-fa7f-4931-ac1d-c51f83824177-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "fc436136-fa7f-4931-ac1d-c51f83824177" (UID: "fc436136-fa7f-4931-ac1d-c51f83824177"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:36:37.707951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.707921 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2rgn\" (UniqueName: \"kubernetes.io/projected/fc436136-fa7f-4931-ac1d-c51f83824177-kube-api-access-r2rgn\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:36:37.707951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.707950 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgdvt\" (UniqueName: \"kubernetes.io/projected/8b373889-0841-4fac-a1ef-41a9545e76e4-kube-api-access-wgdvt\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:36:37.708172 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:37.707962 2582 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/fc436136-fa7f-4931-ac1d-c51f83824177-postgresdata\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:36:38.379481 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:38.379454 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" Jul 29 08:36:38.379937 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:38.379451 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-5r2vs" event={"ID":"fc436136-fa7f-4931-ac1d-c51f83824177","Type":"ContainerDied","Data":"688bee981ad2d2671b94a2a3660f38e0c83c2b186174a004b817b37ff4a4ecff"} Jul 29 08:36:38.379937 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:38.379587 2582 scope.go:117] "RemoveContainer" containerID="e2deb87263f8fb75cc77fdee8f44a472f7c919b5c429aa03b5a4388805ba1c4f" Jul 29 08:36:38.381047 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:38.381028 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" event={"ID":"8b373889-0841-4fac-a1ef-41a9545e76e4","Type":"ContainerDied","Data":"69426aa530828130b93c9eee79f59d7c3f2f59aadc5f4df8ee9bbed47ff1ad64"} Jul 29 08:36:38.381047 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:38.381049 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-9lpff" Jul 29 08:36:38.388691 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:38.388674 2582 scope.go:117] "RemoveContainer" containerID="8c9baae00cc3b7a33b4d03e2b85e25a356df39a3ef98c1f57c88a5f8a6be4237" Jul 29 08:36:38.398444 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:38.398423 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5r2vs"] Jul 29 08:36:38.401573 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:38.401555 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-5r2vs"] Jul 29 08:36:38.412489 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:38.412470 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-9lpff"] Jul 29 08:36:38.418471 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:38.418450 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-9lpff"] Jul 29 08:36:40.089552 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:40.089513 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:40.183382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:40.183347 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:40.250804 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:40.250771 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b373889-0841-4fac-a1ef-41a9545e76e4" path="/var/lib/kubelet/pods/8b373889-0841-4fac-a1ef-41a9545e76e4/volumes" Jul 29 08:36:40.251317 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:40.251300 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc436136-fa7f-4931-ac1d-c51f83824177" path="/var/lib/kubelet/pods/fc436136-fa7f-4931-ac1d-c51f83824177/volumes" Jul 29 08:36:43.099369 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:43.099332 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:43.099830 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:43.099387 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:43.104251 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:43.104227 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:43.409340 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:43.409255 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-648d8678b4-dg7bw" Jul 29 08:36:43.463889 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:43.463855 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6975ffc864-nq5t5"] Jul 29 08:36:43.577313 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:43.577283 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:43.678917 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:43.678829 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:43.741319 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:43.741290 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:44.064934 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:44.064893 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 29 08:36:45.216462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:45.216425 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:45.314792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:45.314756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:45.778753 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:45.778719 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:36:46.597551 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:46.597509 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Jul 29 08:36:54.065475 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:54.065432 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 29 08:36:56.597958 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:56.597915 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Jul 29 08:36:56.598439 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:36:56.598023 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:37:02.087577 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:02.087534 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:02.150724 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:02.150693 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:04.065321 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:04.065276 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 29 08:37:04.065749 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:04.065423 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:37:05.508191 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.508154 2582 generic.go:356] "Generic (PLEG): container finished" podID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerID="71185af92b5b654cb15b273a7a5292dcafd1d3a822efc03bd8db92304c56045f" exitCode=137 Jul 29 08:37:05.508657 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.508222 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" event={"ID":"be0fc037-bbcb-401a-87a2-489c5bee7e31","Type":"ContainerDied","Data":"71185af92b5b654cb15b273a7a5292dcafd1d3a822efc03bd8db92304c56045f"} Jul 29 08:37:05.510478 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.510457 2582 generic.go:356] "Generic (PLEG): container finished" podID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerID="6e1339bcee8ac31d7d600ae682c816af383e3f8602ce5f813129da0987bcf778" exitCode=137 Jul 29 08:37:05.510586 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.510531 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" event={"ID":"26ec9b5d-9658-47a8-857f-8f210bc83d13","Type":"ContainerDied","Data":"6e1339bcee8ac31d7d600ae682c816af383e3f8602ce5f813129da0987bcf778"} Jul 29 08:37:05.721898 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.721874 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:37:05.752082 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.752058 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:37:05.854057 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.853965 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be0fc037-bbcb-401a-87a2-489c5bee7e31-feast-data\") pod \"be0fc037-bbcb-401a-87a2-489c5bee7e31\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " Jul 29 08:37:05.854057 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.853998 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbwfc\" (UniqueName: \"kubernetes.io/projected/26ec9b5d-9658-47a8-857f-8f210bc83d13-kube-api-access-sbwfc\") pod \"26ec9b5d-9658-47a8-857f-8f210bc83d13\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " Jul 29 08:37:05.854057 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.854020 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-online-tls\") pod \"be0fc037-bbcb-401a-87a2-489c5bee7e31\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " Jul 29 08:37:05.854057 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.854055 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-registry-tls\") pod \"be0fc037-bbcb-401a-87a2-489c5bee7e31\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " Jul 29 08:37:05.854393 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.854101 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-registry-tls\") pod \"26ec9b5d-9658-47a8-857f-8f210bc83d13\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " Jul 29 08:37:05.854393 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.854121 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsm76\" (UniqueName: \"kubernetes.io/projected/be0fc037-bbcb-401a-87a2-489c5bee7e31-kube-api-access-jsm76\") pod \"be0fc037-bbcb-401a-87a2-489c5bee7e31\" (UID: \"be0fc037-bbcb-401a-87a2-489c5bee7e31\") " Jul 29 08:37:05.854393 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.854250 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/26ec9b5d-9658-47a8-857f-8f210bc83d13-feast-data\") pod \"26ec9b5d-9658-47a8-857f-8f210bc83d13\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " Jul 29 08:37:05.854393 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.854311 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-online-tls\") pod \"26ec9b5d-9658-47a8-857f-8f210bc83d13\" (UID: \"26ec9b5d-9658-47a8-857f-8f210bc83d13\") " Jul 29 08:37:05.855200 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.855148 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ec9b5d-9658-47a8-857f-8f210bc83d13-feast-data" (OuterVolumeSpecName: "feast-data") pod "26ec9b5d-9658-47a8-857f-8f210bc83d13" (UID: "26ec9b5d-9658-47a8-857f-8f210bc83d13"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:37:05.856868 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.856839 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-online-tls" (OuterVolumeSpecName: "online-tls") pod "26ec9b5d-9658-47a8-857f-8f210bc83d13" (UID: "26ec9b5d-9658-47a8-857f-8f210bc83d13"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:37:05.857015 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.856993 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-online-tls" (OuterVolumeSpecName: "online-tls") pod "be0fc037-bbcb-401a-87a2-489c5bee7e31" (UID: "be0fc037-bbcb-401a-87a2-489c5bee7e31"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:37:05.857168 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.857149 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "be0fc037-bbcb-401a-87a2-489c5bee7e31" (UID: "be0fc037-bbcb-401a-87a2-489c5bee7e31"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:37:05.857229 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.857182 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "26ec9b5d-9658-47a8-857f-8f210bc83d13" (UID: "26ec9b5d-9658-47a8-857f-8f210bc83d13"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:37:05.857229 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.857183 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0fc037-bbcb-401a-87a2-489c5bee7e31-kube-api-access-jsm76" (OuterVolumeSpecName: "kube-api-access-jsm76") pod "be0fc037-bbcb-401a-87a2-489c5bee7e31" (UID: "be0fc037-bbcb-401a-87a2-489c5bee7e31"). InnerVolumeSpecName "kube-api-access-jsm76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:37:05.857229 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.857211 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ec9b5d-9658-47a8-857f-8f210bc83d13-kube-api-access-sbwfc" (OuterVolumeSpecName: "kube-api-access-sbwfc") pod "26ec9b5d-9658-47a8-857f-8f210bc83d13" (UID: "26ec9b5d-9658-47a8-857f-8f210bc83d13"). InnerVolumeSpecName "kube-api-access-sbwfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:37:05.864188 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.864168 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0fc037-bbcb-401a-87a2-489c5bee7e31-feast-data" (OuterVolumeSpecName: "feast-data") pod "be0fc037-bbcb-401a-87a2-489c5bee7e31" (UID: "be0fc037-bbcb-401a-87a2-489c5bee7e31"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:37:05.955404 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.955352 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:05.955404 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.955397 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:05.955404 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.955408 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsm76\" (UniqueName: \"kubernetes.io/projected/be0fc037-bbcb-401a-87a2-489c5bee7e31-kube-api-access-jsm76\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:05.955404 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.955419 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/26ec9b5d-9658-47a8-857f-8f210bc83d13-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:05.955724 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.955428 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/26ec9b5d-9658-47a8-857f-8f210bc83d13-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:05.955724 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.955435 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/be0fc037-bbcb-401a-87a2-489c5bee7e31-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:05.955724 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.955444 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbwfc\" (UniqueName: \"kubernetes.io/projected/26ec9b5d-9658-47a8-857f-8f210bc83d13-kube-api-access-sbwfc\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:05.955724 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:05.955452 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/be0fc037-bbcb-401a-87a2-489c5bee7e31-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:06.516596 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.516502 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" event={"ID":"be0fc037-bbcb-401a-87a2-489c5bee7e31","Type":"ContainerDied","Data":"6de1b3a56a0d5aacfea4c1dde0d5d61a34041f2294fe4022f0fea819bc102eee"} Jul 29 08:37:06.516596 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.516529 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg" Jul 29 08:37:06.516596 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.516561 2582 scope.go:117] "RemoveContainer" containerID="d91a6b818f02f1dbfd8d21fa20f5b0daf62361c8b77a8fed9206e10408daddfc" Jul 29 08:37:06.519027 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.518997 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" Jul 29 08:37:06.519147 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.518921 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f" event={"ID":"26ec9b5d-9658-47a8-857f-8f210bc83d13","Type":"ContainerDied","Data":"44d33fce0e52cf37820f7a4e797ed34f8b8e8c3b2822bc3870194dabee87994f"} Jul 29 08:37:06.525531 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.525515 2582 scope.go:117] "RemoveContainer" containerID="71185af92b5b654cb15b273a7a5292dcafd1d3a822efc03bd8db92304c56045f" Jul 29 08:37:06.534158 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.534134 2582 scope.go:117] "RemoveContainer" containerID="359e0473d6d87da2afa75421be05d02883afe67f4d5fffd3c8558151e082292f" Jul 29 08:37:06.535408 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.535390 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg"] Jul 29 08:37:06.539942 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.539922 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5859ff5d89-rhbfg"] Jul 29 08:37:06.543202 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.543184 2582 scope.go:117] "RemoveContainer" containerID="0c88a9a386e708a7406819ed5a435da8a75dd93aa35841c78ecfef304a90760e" Jul 29 08:37:06.550810 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.550788 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f"] Jul 29 08:37:06.551689 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.551669 2582 scope.go:117] "RemoveContainer" containerID="faae5951a79b70170e377ad3f30b481e3cf4067f363a009ead5d2ed8a18016a4" Jul 29 08:37:06.554678 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.554657 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5976578f46-k5h7f"] Jul 29 08:37:06.559734 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.559717 2582 scope.go:117] "RemoveContainer" containerID="6e1339bcee8ac31d7d600ae682c816af383e3f8602ce5f813129da0987bcf778" Jul 29 08:37:06.567760 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.567737 2582 scope.go:117] "RemoveContainer" containerID="fec26e3526a7ff4e3bcb41442ab95b326e6fe762ec9cb14b65777b1e58bcce52" Jul 29 08:37:06.574929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:06.574903 2582 scope.go:117] "RemoveContainer" containerID="51683bc89281a31b6df4579bb664429c536d21cedaeaa823459b0e857650c53f" Jul 29 08:37:08.251089 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.251056 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" path="/var/lib/kubelet/pods/26ec9b5d-9658-47a8-857f-8f210bc83d13/volumes" Jul 29 08:37:08.251556 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.251541 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" path="/var/lib/kubelet/pods/be0fc037-bbcb-401a-87a2-489c5bee7e31/volumes" Jul 29 08:37:08.487050 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.486984 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6975ffc864-nq5t5" podUID="8eda6475-9851-48cd-bbda-d62724b9fc1c" containerName="console" containerID="cri-o://7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32" gracePeriod=15 Jul 29 08:37:08.736652 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.736630 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:37:08.881614 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.881517 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-service-ca\") pod \"8eda6475-9851-48cd-bbda-d62724b9fc1c\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " Jul 29 08:37:08.881614 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.881640 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-oauth-serving-cert\") pod \"8eda6475-9851-48cd-bbda-d62724b9fc1c\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " Jul 29 08:37:08.881866 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.881663 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-trusted-ca-bundle\") pod \"8eda6475-9851-48cd-bbda-d62724b9fc1c\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " Jul 29 08:37:08.881866 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.881696 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-serving-cert\") pod \"8eda6475-9851-48cd-bbda-d62724b9fc1c\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " Jul 29 08:37:08.881866 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.881718 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-oauth-config\") pod \"8eda6475-9851-48cd-bbda-d62724b9fc1c\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " Jul 29 08:37:08.881866 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.881734 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-config\") pod \"8eda6475-9851-48cd-bbda-d62724b9fc1c\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " Jul 29 08:37:08.881866 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.881754 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxkml\" (UniqueName: \"kubernetes.io/projected/8eda6475-9851-48cd-bbda-d62724b9fc1c-kube-api-access-rxkml\") pod \"8eda6475-9851-48cd-bbda-d62724b9fc1c\" (UID: \"8eda6475-9851-48cd-bbda-d62724b9fc1c\") " Jul 29 08:37:08.882109 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.881997 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-service-ca" (OuterVolumeSpecName: "service-ca") pod "8eda6475-9851-48cd-bbda-d62724b9fc1c" (UID: "8eda6475-9851-48cd-bbda-d62724b9fc1c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:37:08.882173 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.882083 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8eda6475-9851-48cd-bbda-d62724b9fc1c" (UID: "8eda6475-9851-48cd-bbda-d62724b9fc1c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:37:08.882230 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.882177 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-config" (OuterVolumeSpecName: "console-config") pod "8eda6475-9851-48cd-bbda-d62724b9fc1c" (UID: "8eda6475-9851-48cd-bbda-d62724b9fc1c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:37:08.882286 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.882254 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8eda6475-9851-48cd-bbda-d62724b9fc1c" (UID: "8eda6475-9851-48cd-bbda-d62724b9fc1c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:37:08.884117 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.884088 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8eda6475-9851-48cd-bbda-d62724b9fc1c" (UID: "8eda6475-9851-48cd-bbda-d62724b9fc1c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:37:08.884241 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.884222 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eda6475-9851-48cd-bbda-d62724b9fc1c-kube-api-access-rxkml" (OuterVolumeSpecName: "kube-api-access-rxkml") pod "8eda6475-9851-48cd-bbda-d62724b9fc1c" (UID: "8eda6475-9851-48cd-bbda-d62724b9fc1c"). InnerVolumeSpecName "kube-api-access-rxkml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:37:08.884241 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.884229 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8eda6475-9851-48cd-bbda-d62724b9fc1c" (UID: "8eda6475-9851-48cd-bbda-d62724b9fc1c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:37:08.982831 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.982790 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-serving-cert\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:08.982831 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.982824 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-oauth-config\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:08.982831 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.982834 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-console-config\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:08.982831 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.982843 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxkml\" (UniqueName: \"kubernetes.io/projected/8eda6475-9851-48cd-bbda-d62724b9fc1c-kube-api-access-rxkml\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:08.983102 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.982853 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-service-ca\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:08.983102 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.982861 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-oauth-serving-cert\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:08.983102 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:08.982868 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eda6475-9851-48cd-bbda-d62724b9fc1c-trusted-ca-bundle\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:37:09.536982 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:09.536939 2582 generic.go:356] "Generic (PLEG): container finished" podID="8eda6475-9851-48cd-bbda-d62724b9fc1c" containerID="7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32" exitCode=0 Jul 29 08:37:09.537411 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:09.536994 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6975ffc864-nq5t5" Jul 29 08:37:09.537411 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:09.537021 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6975ffc864-nq5t5" event={"ID":"8eda6475-9851-48cd-bbda-d62724b9fc1c","Type":"ContainerDied","Data":"7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32"} Jul 29 08:37:09.537411 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:09.537108 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6975ffc864-nq5t5" event={"ID":"8eda6475-9851-48cd-bbda-d62724b9fc1c","Type":"ContainerDied","Data":"ef30afccc759c02cd82edb78cc584e8e5f23a2068aedcb2388c377d5839ce3d5"} Jul 29 08:37:09.537411 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:09.537133 2582 scope.go:117] "RemoveContainer" containerID="7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32" Jul 29 08:37:09.546612 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:09.546592 2582 scope.go:117] "RemoveContainer" containerID="7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32" Jul 29 08:37:09.546910 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:37:09.546891 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32\": container with ID starting with 7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32 not found: ID does not exist" containerID="7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32" Jul 29 08:37:09.546969 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:09.546920 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32"} err="failed to get container status \"7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32\": rpc error: code = NotFound desc = could not find container \"7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32\": container with ID starting with 7fe3f6c38eb5b16be69307279e5944e797ad3446aa9fac5643302a76d631ac32 not found: ID does not exist" Jul 29 08:37:09.557962 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:09.557935 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6975ffc864-nq5t5"] Jul 29 08:37:09.561654 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:09.561613 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6975ffc864-nq5t5"] Jul 29 08:37:10.251154 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:10.251119 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eda6475-9851-48cd-bbda-d62724b9fc1c" path="/var/lib/kubelet/pods/8eda6475-9851-48cd-bbda-d62724b9fc1c/volumes" Jul 29 08:37:14.374934 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:14.374907 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:14.478238 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:14.478204 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:15.797704 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:15.797669 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:23.226693 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.226594 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:23.329226 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.329195 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:23.380422 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380388 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw"] Jul 29 08:37:23.380720 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380707 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="feast-init" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380722 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="feast-init" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380733 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="feast-apply" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380738 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="feast-apply" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380748 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b373889-0841-4fac-a1ef-41a9545e76e4" containerName="redis" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380754 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b373889-0841-4fac-a1ef-41a9545e76e4" containerName="redis" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380760 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eda6475-9851-48cd-bbda-d62724b9fc1c" containerName="console" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380765 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eda6475-9851-48cd-bbda-d62724b9fc1c" containerName="console" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380770 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="feast-init" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380777 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="feast-init" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380786 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d950279-670c-4203-94ae-46c922d2b1f6" containerName="manager" Jul 29 08:37:23.380792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380793 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d950279-670c-4203-94ae-46c922d2b1f6" containerName="manager" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380806 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="online" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380812 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="online" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380824 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="feast-apply" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380829 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="feast-apply" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380835 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="registry" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380839 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="registry" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380845 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc436136-fa7f-4931-ac1d-c51f83824177" containerName="postgres" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380850 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc436136-fa7f-4931-ac1d-c51f83824177" containerName="postgres" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380855 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="registry" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380863 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="registry" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380877 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="online" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380881 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="online" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380943 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eda6475-9851-48cd-bbda-d62724b9fc1c" containerName="console" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380950 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="registry" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380955 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0fc037-bbcb-401a-87a2-489c5bee7e31" containerName="online" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380962 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="registry" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380967 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d950279-670c-4203-94ae-46c922d2b1f6" containerName="manager" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380976 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ec9b5d-9658-47a8-857f-8f210bc83d13" containerName="online" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380982 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc436136-fa7f-4931-ac1d-c51f83824177" containerName="postgres" Jul 29 08:37:23.381201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.380987 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b373889-0841-4fac-a1ef-41a9545e76e4" containerName="redis" Jul 29 08:37:23.384582 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.384565 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.387469 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.387448 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 29 08:37:23.387469 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.387456 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 29 08:37:23.387672 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.387459 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 29 08:37:23.387826 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.387811 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 29 08:37:23.387885 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.387830 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-2s9gz\"" Jul 29 08:37:23.387932 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.387813 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 29 08:37:23.388141 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.388129 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 29 08:37:23.399675 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.399651 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw"] Jul 29 08:37:23.405871 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.405848 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:23.475349 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.475319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:23.510442 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.510370 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1fd8d8d6-7e19-4355-81de-f0d64f639b77-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.510442 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.510409 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7t8c\" (UniqueName: \"kubernetes.io/projected/1fd8d8d6-7e19-4355-81de-f0d64f639b77-kube-api-access-v7t8c\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.510442 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.510438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.510682 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.510498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.510682 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.510547 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.510682 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.510569 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.573981 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.573927 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:23.611760 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.611733 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.611909 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.611763 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.611909 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.611796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.611999 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.611913 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.611999 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.611976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1fd8d8d6-7e19-4355-81de-f0d64f639b77-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.612110 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.612011 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7t8c\" (UniqueName: \"kubernetes.io/projected/1fd8d8d6-7e19-4355-81de-f0d64f639b77-kube-api-access-v7t8c\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.612382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.612351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1fd8d8d6-7e19-4355-81de-f0d64f639b77-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.614459 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.614429 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.614548 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.614465 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.614731 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.614716 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.614778 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.614762 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.621152 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.621132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7t8c\" (UniqueName: \"kubernetes.io/projected/1fd8d8d6-7e19-4355-81de-f0d64f639b77-kube-api-access-v7t8c\") pod \"feast-simple-feast-setup-777d6cc998-ppqvw\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.672998 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.672973 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:37:23.708197 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.708176 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:23.849557 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.849530 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw"] Jul 29 08:37:23.850558 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:37:23.850535 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fd8d8d6_7e19_4355_81de_f0d64f639b77.slice/crio-f2b4ec5ca0e03fb9c9d2b68d74af9aee4ab7ecf5b1cfdfa29bdb0d3483f8c48b WatchSource:0}: Error finding container f2b4ec5ca0e03fb9c9d2b68d74af9aee4ab7ecf5b1cfdfa29bdb0d3483f8c48b: Status 404 returned error can't find the container with id f2b4ec5ca0e03fb9c9d2b68d74af9aee4ab7ecf5b1cfdfa29bdb0d3483f8c48b Jul 29 08:37:23.852394 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:23.852377 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:37:24.600807 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:24.600775 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerStarted","Data":"f2b4ec5ca0e03fb9c9d2b68d74af9aee4ab7ecf5b1cfdfa29bdb0d3483f8c48b"} Jul 29 08:37:38.672011 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:38.671976 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerStarted","Data":"4f7e4126f9c0790d4a9bd1897aa8405bc82539c47601dc678f2e1aa7956ed5e5"} Jul 29 08:37:43.698058 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:43.698023 2582 generic.go:356] "Generic (PLEG): container finished" podID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerID="4f7e4126f9c0790d4a9bd1897aa8405bc82539c47601dc678f2e1aa7956ed5e5" exitCode=0 Jul 29 08:37:43.698549 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:43.698103 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerDied","Data":"4f7e4126f9c0790d4a9bd1897aa8405bc82539c47601dc678f2e1aa7956ed5e5"} Jul 29 08:37:44.704235 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:44.704187 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerStarted","Data":"22c60623089dde5cbfa29652bcc2a3aa7932245ea4ef292bfd7a0d3ca58dbb83"} Jul 29 08:37:49.726821 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:49.726782 2582 generic.go:356] "Generic (PLEG): container finished" podID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerID="22c60623089dde5cbfa29652bcc2a3aa7932245ea4ef292bfd7a0d3ca58dbb83" exitCode=0 Jul 29 08:37:49.727245 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:49.726858 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerDied","Data":"22c60623089dde5cbfa29652bcc2a3aa7932245ea4ef292bfd7a0d3ca58dbb83"} Jul 29 08:37:50.738688 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:50.738645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerStarted","Data":"f4e7f15db23c04053ce8f89d22651a77d8623ec90b698f15e05b15456ec620ff"} Jul 29 08:37:50.739195 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:50.738713 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerStarted","Data":"d2004a5d40af627408e53df53cb5c661b93f6230dd5d6d00bacc9e25021a767b"} Jul 29 08:37:50.739195 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:50.738732 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerStarted","Data":"ace160ab2b582904f0210c19e1c49e3d1cf3427b1bd13bbd6986b5994643b171"} Jul 29 08:37:50.739195 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:50.738750 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerStarted","Data":"f010970d6a99331a447569933a26c3d88baa71caafd227cf0641df5ada2270f5"} Jul 29 08:37:50.769710 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:50.769651 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podStartSLOduration=13.597899002 podStartE2EDuration="27.769603373s" totalImagesPullingTime="14.171704371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:37:50.767700831 +0000 UTC m=+1191.080881692" watchObservedRunningTime="2026-07-29 08:37:50.769603373 +0000 UTC m=+1191.082784233" Jul 29 08:37:53.709338 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.709284 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:53.709843 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.709383 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:53.709843 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.709423 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:53.709843 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.709443 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:53.709843 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.709461 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:53.709843 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.709495 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:53.709843 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.709512 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:53.709843 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.709529 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:53.711307 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.711270 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.38:6571: connect: connection refused" Jul 29 08:37:53.711432 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.711318 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Jul 29 08:37:53.711432 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.711367 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.38:8816: connect: connection refused" Jul 29 08:37:53.711432 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:53.711325 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused" Jul 29 08:37:56.708962 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:56.708905 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Jul 29 08:37:56.709510 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:56.708961 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused" Jul 29 08:37:56.709510 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:56.709224 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.38:6571: connect: connection refused" Jul 29 08:37:56.709677 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:56.709610 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.38:8816: connect: connection refused" Jul 29 08:37:59.710301 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:59.710270 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:59.710766 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:59.710390 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:59.710766 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:59.710643 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:59.711161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:59.711134 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:59.711264 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:59.711201 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:59.711264 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:59.711233 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:59.715330 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:59.715307 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:37:59.818275 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:37:59.818245 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:39:01.735699 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:01.735659 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw"] Jul 29 08:39:01.736228 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:01.735960 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="registry" containerID="cri-o://f010970d6a99331a447569933a26c3d88baa71caafd227cf0641df5ada2270f5" gracePeriod=30 Jul 29 08:39:01.736228 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:01.736102 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="online" containerID="cri-o://ace160ab2b582904f0210c19e1c49e3d1cf3427b1bd13bbd6986b5994643b171" gracePeriod=30 Jul 29 08:39:01.736228 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:01.736125 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="offline" containerID="cri-o://d2004a5d40af627408e53df53cb5c661b93f6230dd5d6d00bacc9e25021a767b" gracePeriod=30 Jul 29 08:39:01.736415 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:01.736275 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="ui" containerID="cri-o://f4e7f15db23c04053ce8f89d22651a77d8623ec90b698f15e05b15456ec620ff" gracePeriod=30 Jul 29 08:39:01.784079 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:01.784041 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:39:01.841266 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:01.841226 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:39:01.946999 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:01.946960 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:39:02.048819 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.048729 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:39:02.079916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.079882 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx"] Jul 29 08:39:02.083608 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.083587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.086146 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.086117 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-gx7cv\"" Jul 29 08:39:02.096417 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.096393 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx"] Jul 29 08:39:02.126987 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.126950 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:39:02.176026 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.175987 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/af073fc2-3744-4292-b7e2-41fac11eac8f-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.176026 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.176027 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rmlh\" (UniqueName: \"kubernetes.io/projected/af073fc2-3744-4292-b7e2-41fac11eac8f-kube-api-access-6rmlh\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.176242 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.176159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.176242 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.176196 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.176310 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.176246 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.176310 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.176286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.185917 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.185886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:39:02.248391 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.248356 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:39:02.277353 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.277310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.277353 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.277353 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.277657 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.277500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.277657 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.277556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.277657 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.277641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/af073fc2-3744-4292-b7e2-41fac11eac8f-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.277830 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.277672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rmlh\" (UniqueName: \"kubernetes.io/projected/af073fc2-3744-4292-b7e2-41fac11eac8f-kube-api-access-6rmlh\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.278077 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.278039 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/af073fc2-3744-4292-b7e2-41fac11eac8f-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.280531 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.280507 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.280664 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.280508 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.280664 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.280614 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.280851 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.280829 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.286316 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.286291 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rmlh\" (UniqueName: \"kubernetes.io/projected/af073fc2-3744-4292-b7e2-41fac11eac8f-kube-api-access-6rmlh\") pod \"feast-simple-feast-setup-777d6cc998-9mpwx\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.348811 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.348781 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:39:02.393742 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.393703 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:02.539994 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:02.539878 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx"] Jul 29 08:39:02.542133 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:39:02.542107 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf073fc2_3744_4292_b7e2_41fac11eac8f.slice/crio-810cbabbbf76e7bd4fc1447f5b5e4f9d1907c87895141e4183854fc70ae4e55a WatchSource:0}: Error finding container 810cbabbbf76e7bd4fc1447f5b5e4f9d1907c87895141e4183854fc70ae4e55a: Status 404 returned error can't find the container with id 810cbabbbf76e7bd4fc1447f5b5e4f9d1907c87895141e4183854fc70ae4e55a Jul 29 08:39:03.068330 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:03.068292 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerStarted","Data":"369db23630bf27323cbc92648bdd0c8b93b52c5b8f38dbf6454d2d1aacd2793c"} Jul 29 08:39:03.068812 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:03.068343 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerStarted","Data":"810cbabbbf76e7bd4fc1447f5b5e4f9d1907c87895141e4183854fc70ae4e55a"} Jul 29 08:39:03.071735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:03.071709 2582 generic.go:356] "Generic (PLEG): container finished" podID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerID="f4e7f15db23c04053ce8f89d22651a77d8623ec90b698f15e05b15456ec620ff" exitCode=0 Jul 29 08:39:03.071735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:03.071730 2582 generic.go:356] "Generic (PLEG): container finished" podID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerID="ace160ab2b582904f0210c19e1c49e3d1cf3427b1bd13bbd6986b5994643b171" exitCode=0 Jul 29 08:39:03.071896 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:03.071767 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerDied","Data":"f4e7f15db23c04053ce8f89d22651a77d8623ec90b698f15e05b15456ec620ff"} Jul 29 08:39:03.071896 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:03.071792 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerDied","Data":"ace160ab2b582904f0210c19e1c49e3d1cf3427b1bd13bbd6986b5994643b171"} Jul 29 08:39:08.094674 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:08.094609 2582 generic.go:356] "Generic (PLEG): container finished" podID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerID="369db23630bf27323cbc92648bdd0c8b93b52c5b8f38dbf6454d2d1aacd2793c" exitCode=0 Jul 29 08:39:08.095116 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:08.094693 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerDied","Data":"369db23630bf27323cbc92648bdd0c8b93b52c5b8f38dbf6454d2d1aacd2793c"} Jul 29 08:39:09.101010 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:09.100975 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerStarted","Data":"883d2dd4e60c7df78604ba8e2cc4e4b5dc4da8b625676a5b487d8efd865ea887"} Jul 29 08:39:09.711499 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:09.711446 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Jul 29 08:39:09.813144 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:09.813099 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused" Jul 29 08:39:11.269325 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:11.269289 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:39:11.335100 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:11.335055 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:39:14.125045 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:14.124958 2582 generic.go:356] "Generic (PLEG): container finished" podID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerID="883d2dd4e60c7df78604ba8e2cc4e4b5dc4da8b625676a5b487d8efd865ea887" exitCode=0 Jul 29 08:39:14.125045 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:14.125026 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerDied","Data":"883d2dd4e60c7df78604ba8e2cc4e4b5dc4da8b625676a5b487d8efd865ea887"} Jul 29 08:39:15.135647 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:15.135583 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerStarted","Data":"6367967cce016f8775d3474b6a3c166adf114990406d4044d04a47e38fbcc8d7"} Jul 29 08:39:15.136142 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:15.135668 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerStarted","Data":"c9233db724f859a386db417dce301c80ebbc0ec38e469fbf3f7af54f194cd878"} Jul 29 08:39:15.136142 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:15.135688 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerStarted","Data":"8f89712b1279673492f30d4e8112d46d450f65754fa1ca707ca2f2a6884aabf0"} Jul 29 08:39:15.136142 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:15.135705 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerStarted","Data":"5751c237edf5bb4ba60ea5a05bac15e7bcd1d960de8c053682a9c3ecdd1fe8ec"} Jul 29 08:39:15.161265 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:15.161195 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podStartSLOduration=13.161171048 podStartE2EDuration="13.161171048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:39:15.159601988 +0000 UTC m=+1275.472782848" watchObservedRunningTime="2026-07-29 08:39:15.161171048 +0000 UTC m=+1275.474351909" Jul 29 08:39:17.394837 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:17.394789 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:17.395317 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:17.394873 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:17.395317 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:17.394893 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:17.395317 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:17.394910 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:17.396753 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:17.396715 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused" Jul 29 08:39:17.396883 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:17.396759 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused" Jul 29 08:39:17.396883 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:17.396800 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.39:8816: connect: connection refused" Jul 29 08:39:17.396883 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:17.396754 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.39:6571: connect: connection refused" Jul 29 08:39:19.711162 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:19.711114 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Jul 29 08:39:19.812416 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:19.812364 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused" Jul 29 08:39:20.394525 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:20.394296 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused" Jul 29 08:39:20.395100 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:20.395041 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:20.395253 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:20.395232 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:20.395812 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:20.395529 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:20.395812 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:20.395567 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:20.395812 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:20.395737 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:20.396037 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:20.395988 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:20.400469 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:20.400451 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:21.172233 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:21.172208 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:21.175670 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:21.175650 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:22.394529 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:22.394485 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:23.394895 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:23.394865 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:24.186204 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:24.186175 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:39:29.711763 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:29.711700 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.38:8443: connect: connection refused" Jul 29 08:39:29.712248 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:29.711884 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:39:29.812548 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:29.812502 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="online" probeResult="failure" output="Get \"https://10.134.0.38:6567/health\": dial tcp 10.134.0.38:6567: connect: connection refused" Jul 29 08:39:29.812780 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:29.812652 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:39:32.224731 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.224694 2582 generic.go:356] "Generic (PLEG): container finished" podID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerID="d2004a5d40af627408e53df53cb5c661b93f6230dd5d6d00bacc9e25021a767b" exitCode=137 Jul 29 08:39:32.224731 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.224724 2582 generic.go:356] "Generic (PLEG): container finished" podID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerID="f010970d6a99331a447569933a26c3d88baa71caafd227cf0641df5ada2270f5" exitCode=137 Jul 29 08:39:32.225142 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.224763 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerDied","Data":"d2004a5d40af627408e53df53cb5c661b93f6230dd5d6d00bacc9e25021a767b"} Jul 29 08:39:32.225142 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.224811 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerDied","Data":"f010970d6a99331a447569933a26c3d88baa71caafd227cf0641df5ada2270f5"} Jul 29 08:39:32.386867 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.386845 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:39:32.429612 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.429585 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1fd8d8d6-7e19-4355-81de-f0d64f639b77-feast-data\") pod \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " Jul 29 08:39:32.429612 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.429614 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7t8c\" (UniqueName: \"kubernetes.io/projected/1fd8d8d6-7e19-4355-81de-f0d64f639b77-kube-api-access-v7t8c\") pod \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " Jul 29 08:39:32.429857 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.429690 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-registry-tls\") pod \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " Jul 29 08:39:32.429857 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.429709 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-online-tls\") pod \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " Jul 29 08:39:32.429857 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.429723 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-offline-tls\") pod \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " Jul 29 08:39:32.429857 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.429768 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-ui-tls\") pod \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\" (UID: \"1fd8d8d6-7e19-4355-81de-f0d64f639b77\") " Jul 29 08:39:32.430201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.430157 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd8d8d6-7e19-4355-81de-f0d64f639b77-feast-data" (OuterVolumeSpecName: "feast-data") pod "1fd8d8d6-7e19-4355-81de-f0d64f639b77" (UID: "1fd8d8d6-7e19-4355-81de-f0d64f639b77"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:39:32.432160 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.432136 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1fd8d8d6-7e19-4355-81de-f0d64f639b77" (UID: "1fd8d8d6-7e19-4355-81de-f0d64f639b77"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:39:32.432404 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.432385 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd8d8d6-7e19-4355-81de-f0d64f639b77-kube-api-access-v7t8c" (OuterVolumeSpecName: "kube-api-access-v7t8c") pod "1fd8d8d6-7e19-4355-81de-f0d64f639b77" (UID: "1fd8d8d6-7e19-4355-81de-f0d64f639b77"). InnerVolumeSpecName "kube-api-access-v7t8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:39:32.432740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.432712 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-online-tls" (OuterVolumeSpecName: "online-tls") pod "1fd8d8d6-7e19-4355-81de-f0d64f639b77" (UID: "1fd8d8d6-7e19-4355-81de-f0d64f639b77"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:39:32.432740 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.432726 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1fd8d8d6-7e19-4355-81de-f0d64f639b77" (UID: "1fd8d8d6-7e19-4355-81de-f0d64f639b77"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:39:32.432848 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.432748 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1fd8d8d6-7e19-4355-81de-f0d64f639b77" (UID: "1fd8d8d6-7e19-4355-81de-f0d64f639b77"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:39:32.531130 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.531035 2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-ui-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:39:32.531130 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.531069 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1fd8d8d6-7e19-4355-81de-f0d64f639b77-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:39:32.531130 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.531084 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7t8c\" (UniqueName: \"kubernetes.io/projected/1fd8d8d6-7e19-4355-81de-f0d64f639b77-kube-api-access-v7t8c\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:39:32.531130 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.531097 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:39:32.531130 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.531109 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:39:32.531130 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:32.531120 2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1fd8d8d6-7e19-4355-81de-f0d64f639b77-offline-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:39:33.232032 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:33.232003 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" Jul 29 08:39:33.232484 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:33.232003 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw" event={"ID":"1fd8d8d6-7e19-4355-81de-f0d64f639b77","Type":"ContainerDied","Data":"f2b4ec5ca0e03fb9c9d2b68d74af9aee4ab7ecf5b1cfdfa29bdb0d3483f8c48b"} Jul 29 08:39:33.232484 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:33.232140 2582 scope.go:117] "RemoveContainer" containerID="f4e7f15db23c04053ce8f89d22651a77d8623ec90b698f15e05b15456ec620ff" Jul 29 08:39:33.242092 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:33.242078 2582 scope.go:117] "RemoveContainer" containerID="d2004a5d40af627408e53df53cb5c661b93f6230dd5d6d00bacc9e25021a767b" Jul 29 08:39:33.250096 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:33.250080 2582 scope.go:117] "RemoveContainer" containerID="ace160ab2b582904f0210c19e1c49e3d1cf3427b1bd13bbd6986b5994643b171" Jul 29 08:39:33.256174 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:33.256151 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw"] Jul 29 08:39:33.258468 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:33.258448 2582 scope.go:117] "RemoveContainer" containerID="f010970d6a99331a447569933a26c3d88baa71caafd227cf0641df5ada2270f5" Jul 29 08:39:33.260440 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:33.260421 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-ppqvw"] Jul 29 08:39:33.266380 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:33.266358 2582 scope.go:117] "RemoveContainer" containerID="22c60623089dde5cbfa29652bcc2a3aa7932245ea4ef292bfd7a0d3ca58dbb83" Jul 29 08:39:33.273652 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:33.273638 2582 scope.go:117] "RemoveContainer" containerID="4f7e4126f9c0790d4a9bd1897aa8405bc82539c47601dc678f2e1aa7956ed5e5" Jul 29 08:39:34.251391 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:39:34.251358 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" path="/var/lib/kubelet/pods/1fd8d8d6-7e19-4355-81de-f0d64f639b77/volumes" Jul 29 08:40:11.438262 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:11.438223 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:40:25.636145 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:25.636043 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:40:25.738138 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:25.738103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:40:25.957374 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:25.957339 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:40:26.058539 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.058497 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:40:26.153345 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.153318 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:40:26.157591 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157570 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp"] Jul 29 08:40:26.157909 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157896 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="online" Jul 29 08:40:26.157953 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157912 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="online" Jul 29 08:40:26.157953 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157920 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="ui" Jul 29 08:40:26.157953 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157926 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="ui" Jul 29 08:40:26.157953 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157934 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="feast-apply" Jul 29 08:40:26.157953 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157939 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="feast-apply" Jul 29 08:40:26.157953 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157949 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="offline" Jul 29 08:40:26.157953 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157953 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="offline" Jul 29 08:40:26.158161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157970 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="feast-init" Jul 29 08:40:26.158161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157975 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="feast-init" Jul 29 08:40:26.158161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157980 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="registry" Jul 29 08:40:26.158161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.157984 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="registry" Jul 29 08:40:26.158161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.158028 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="registry" Jul 29 08:40:26.158161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.158034 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="offline" Jul 29 08:40:26.158161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.158044 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="ui" Jul 29 08:40:26.158161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.158050 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fd8d8d6-7e19-4355-81de-f0d64f639b77" containerName="online" Jul 29 08:40:26.162828 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.162812 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.165679 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.165651 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 29 08:40:26.165799 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.165652 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 29 08:40:26.165799 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.165702 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 29 08:40:26.165799 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.165740 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 29 08:40:26.166008 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.165993 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 29 08:40:26.166075 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.166029 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-nq299\"" Jul 29 08:40:26.166303 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.166287 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 29 08:40:26.172700 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.172680 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp"] Jul 29 08:40:26.175670 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.175648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02d10c45-adf2-4e21-8140-f3d08b6aaa65-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.175783 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.175677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/02d10c45-adf2-4e21-8140-f3d08b6aaa65-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.175783 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.175703 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.175909 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.175790 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.175909 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.175830 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.176018 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.175941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d987c\" (UniqueName: \"kubernetes.io/projected/02d10c45-adf2-4e21-8140-f3d08b6aaa65-kube-api-access-d987c\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.212251 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.212172 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:40:26.276801 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.276767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.276978 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.276808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.276978 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:40:26.276926 2582 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jul 29 08:40:26.277072 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:40:26.276982 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-online-tls podName:02d10c45-adf2-4e21-8140-f3d08b6aaa65 nodeName:}" failed. No retries permitted until 2026-07-29 08:40:26.776964041 +0000 UTC m=+1347.090144879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-online-tls") pod "feast-simple-feast-remote-setup-db6955c6c-wmntp" (UID: "02d10c45-adf2-4e21-8140-f3d08b6aaa65") : secret "feast-simple-feast-remote-setup-online-tls" not found Jul 29 08:40:26.277293 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.277255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d987c\" (UniqueName: \"kubernetes.io/projected/02d10c45-adf2-4e21-8140-f3d08b6aaa65-kube-api-access-d987c\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.277439 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.277381 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02d10c45-adf2-4e21-8140-f3d08b6aaa65-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.277525 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.277439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/02d10c45-adf2-4e21-8140-f3d08b6aaa65-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.277525 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.277482 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.277951 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.277926 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02d10c45-adf2-4e21-8140-f3d08b6aaa65-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.278097 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.278079 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/02d10c45-adf2-4e21-8140-f3d08b6aaa65-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.279685 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.279662 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.280028 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.280008 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.284941 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.284925 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d987c\" (UniqueName: \"kubernetes.io/projected/02d10c45-adf2-4e21-8140-f3d08b6aaa65-kube-api-access-d987c\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.461820 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.461784 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:40:26.782916 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.782864 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:26.785507 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:26.785484 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-wmntp\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:27.073360 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:27.073261 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:27.209681 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:27.209474 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp"] Jul 29 08:40:27.212324 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:40:27.212285 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02d10c45_adf2_4e21_8140_f3d08b6aaa65.slice/crio-6ae39c3000a8677f16c8cf9870c77ac02c6189ca4ab1fa2f0d2c10fa2e0eedcc WatchSource:0}: Error finding container 6ae39c3000a8677f16c8cf9870c77ac02c6189ca4ab1fa2f0d2c10fa2e0eedcc: Status 404 returned error can't find the container with id 6ae39c3000a8677f16c8cf9870c77ac02c6189ca4ab1fa2f0d2c10fa2e0eedcc Jul 29 08:40:27.461493 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:27.461458 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerStarted","Data":"1b1629a6c0b7bf5bf146287ca4ea27d5276c2a86a05f73150431efff4fc908e5"} Jul 29 08:40:27.461701 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:27.461508 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerStarted","Data":"6ae39c3000a8677f16c8cf9870c77ac02c6189ca4ab1fa2f0d2c10fa2e0eedcc"} Jul 29 08:40:32.485774 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:32.485736 2582 generic.go:356] "Generic (PLEG): container finished" podID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerID="1b1629a6c0b7bf5bf146287ca4ea27d5276c2a86a05f73150431efff4fc908e5" exitCode=0 Jul 29 08:40:32.485774 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:32.485776 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerDied","Data":"1b1629a6c0b7bf5bf146287ca4ea27d5276c2a86a05f73150431efff4fc908e5"} Jul 29 08:40:33.491494 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:33.491453 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerStarted","Data":"60490df88dc99c4e5fe53bc7bcb1ee59b1f0572461e1290048611328157c41c1"} Jul 29 08:40:38.515199 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:38.515166 2582 generic.go:356] "Generic (PLEG): container finished" podID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerID="60490df88dc99c4e5fe53bc7bcb1ee59b1f0572461e1290048611328157c41c1" exitCode=0 Jul 29 08:40:38.515714 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:38.515246 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerDied","Data":"60490df88dc99c4e5fe53bc7bcb1ee59b1f0572461e1290048611328157c41c1"} Jul 29 08:40:39.534537 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:39.534491 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerStarted","Data":"039d4fbb029aad335baeda8644364c2f352ac8227eab641a52026b6b838e05d0"} Jul 29 08:40:39.535053 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:39.534558 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerStarted","Data":"93b3e96f913cf1b68c475573ac28e4a4427353f222ee61618421c16eb81153d6"} Jul 29 08:40:39.535053 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:39.534580 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerStarted","Data":"9e7b2f359668ac1759040d8fd7ccc7da20c055141e250317adc721a9de6e110c"} Jul 29 08:40:42.074081 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:42.074037 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:42.074523 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:42.074109 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:42.074523 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:42.074129 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:42.076152 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:42.076117 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.40:8816: connect: connection refused" Jul 29 08:40:42.076277 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:42.076145 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused" Jul 29 08:40:42.076390 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:42.076354 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused" Jul 29 08:40:45.074875 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:45.074844 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:45.075356 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:45.074977 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:45.075504 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:45.075351 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:45.075504 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:45.075402 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:45.075673 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:45.075658 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:45.075887 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:45.075868 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:45.081355 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:45.081333 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:45.099059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:45.099004 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podStartSLOduration=19.09898741 podStartE2EDuration="19.09898741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:40:39.556811653 +0000 UTC m=+1359.869992514" watchObservedRunningTime="2026-07-29 08:40:45.09898741 +0000 UTC m=+1365.412168270" Jul 29 08:40:45.562642 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:45.562589 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:45.567322 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:45.567292 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:40:56.057130 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:56.057075 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:40:56.110840 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:40:56.110807 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:41:13.788973 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:13.788935 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:41:47.527396 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.527361 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp"] Jul 29 08:41:47.528137 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.527573 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="online" containerID="cri-o://9e7b2f359668ac1759040d8fd7ccc7da20c055141e250317adc721a9de6e110c" gracePeriod=30 Jul 29 08:41:47.528137 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.527726 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:41:47.528137 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.527789 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="ui" containerID="cri-o://039d4fbb029aad335baeda8644364c2f352ac8227eab641a52026b6b838e05d0" gracePeriod=30 Jul 29 08:41:47.528137 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.527889 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="offline" containerID="cri-o://93b3e96f913cf1b68c475573ac28e4a4427353f222ee61618421c16eb81153d6" gracePeriod=30 Jul 29 08:41:47.598888 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.598852 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:41:47.732528 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.732482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:41:47.752646 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.752592 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx"] Jul 29 08:41:47.752923 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.752898 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="registry" containerID="cri-o://5751c237edf5bb4ba60ea5a05bac15e7bcd1d960de8c053682a9c3ecdd1fe8ec" gracePeriod=30 Jul 29 08:41:47.753026 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.752996 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="ui" containerID="cri-o://6367967cce016f8775d3474b6a3c166adf114990406d4044d04a47e38fbcc8d7" gracePeriod=30 Jul 29 08:41:47.753082 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.753002 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="offline" containerID="cri-o://c9233db724f859a386db417dce301c80ebbc0ec38e469fbf3f7af54f194cd878" gracePeriod=30 Jul 29 08:41:47.753135 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.753005 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="online" containerID="cri-o://8f89712b1279673492f30d4e8112d46d450f65754fa1ca707ca2f2a6884aabf0" gracePeriod=30 Jul 29 08:41:47.810866 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:47.810768 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:41:48.823873 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:48.823834 2582 generic.go:356] "Generic (PLEG): container finished" podID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerID="6367967cce016f8775d3474b6a3c166adf114990406d4044d04a47e38fbcc8d7" exitCode=0 Jul 29 08:41:48.823873 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:48.823858 2582 generic.go:356] "Generic (PLEG): container finished" podID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerID="8f89712b1279673492f30d4e8112d46d450f65754fa1ca707ca2f2a6884aabf0" exitCode=0 Jul 29 08:41:48.824349 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:48.823906 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerDied","Data":"6367967cce016f8775d3474b6a3c166adf114990406d4044d04a47e38fbcc8d7"} Jul 29 08:41:48.824349 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:48.823952 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerDied","Data":"8f89712b1279673492f30d4e8112d46d450f65754fa1ca707ca2f2a6884aabf0"} Jul 29 08:41:48.826511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:48.826487 2582 generic.go:356] "Generic (PLEG): container finished" podID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerID="039d4fbb029aad335baeda8644364c2f352ac8227eab641a52026b6b838e05d0" exitCode=0 Jul 29 08:41:48.826511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:48.826511 2582 generic.go:356] "Generic (PLEG): container finished" podID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerID="9e7b2f359668ac1759040d8fd7ccc7da20c055141e250317adc721a9de6e110c" exitCode=0 Jul 29 08:41:48.826708 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:48.826539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerDied","Data":"039d4fbb029aad335baeda8644364c2f352ac8227eab641a52026b6b838e05d0"} Jul 29 08:41:48.826708 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:48.826572 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerDied","Data":"9e7b2f359668ac1759040d8fd7ccc7da20c055141e250317adc721a9de6e110c"} Jul 29 08:41:51.173421 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:51.173335 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused" Jul 29 08:41:52.873934 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:52.873895 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:41:54.186351 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:54.186310 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused" Jul 29 08:41:55.075908 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:55.075863 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused" Jul 29 08:41:55.563145 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:41:55.563103 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused" Jul 29 08:42:01.173228 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:01.173179 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused" Jul 29 08:42:04.185913 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:04.185864 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused" Jul 29 08:42:05.075555 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:05.075509 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused" Jul 29 08:42:05.563339 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:05.563297 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused" Jul 29 08:42:11.173319 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:11.173271 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.39:6567/health\": dial tcp 10.134.0.39:6567: connect: connection refused" Jul 29 08:42:11.173832 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:11.173401 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:42:14.185848 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:14.185798 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.39:8443: connect: connection refused" Jul 29 08:42:14.186281 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:14.185957 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:42:15.076143 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:15.076102 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.40:8443: connect: connection refused" Jul 29 08:42:15.076348 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:15.076243 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:42:15.563189 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:15.563149 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="online" probeResult="failure" output="Get \"https://10.134.0.40:6567/health\": dial tcp 10.134.0.40:6567: connect: connection refused" Jul 29 08:42:15.563580 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:15.563260 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:42:17.950791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:17.950758 2582 generic.go:356] "Generic (PLEG): container finished" podID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerID="c9233db724f859a386db417dce301c80ebbc0ec38e469fbf3f7af54f194cd878" exitCode=137 Jul 29 08:42:17.950791 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:17.950785 2582 generic.go:356] "Generic (PLEG): container finished" podID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerID="5751c237edf5bb4ba60ea5a05bac15e7bcd1d960de8c053682a9c3ecdd1fe8ec" exitCode=137 Jul 29 08:42:17.951302 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:17.950840 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerDied","Data":"c9233db724f859a386db417dce301c80ebbc0ec38e469fbf3f7af54f194cd878"} Jul 29 08:42:17.951302 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:17.950892 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerDied","Data":"5751c237edf5bb4ba60ea5a05bac15e7bcd1d960de8c053682a9c3ecdd1fe8ec"} Jul 29 08:42:17.953289 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:17.953269 2582 generic.go:356] "Generic (PLEG): container finished" podID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerID="93b3e96f913cf1b68c475573ac28e4a4427353f222ee61618421c16eb81153d6" exitCode=137 Jul 29 08:42:17.953407 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:17.953319 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerDied","Data":"93b3e96f913cf1b68c475573ac28e4a4427353f222ee61618421c16eb81153d6"} Jul 29 08:42:18.183896 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.183873 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:42:18.227339 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.227299 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02d10c45-adf2-4e21-8140-f3d08b6aaa65-feast-data\") pod \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " Jul 29 08:42:18.227339 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.227332 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-online-tls\") pod \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " Jul 29 08:42:18.227610 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.227368 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/02d10c45-adf2-4e21-8140-f3d08b6aaa65-registry-tls\") pod \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " Jul 29 08:42:18.227610 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.227424 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d987c\" (UniqueName: \"kubernetes.io/projected/02d10c45-adf2-4e21-8140-f3d08b6aaa65-kube-api-access-d987c\") pod \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " Jul 29 08:42:18.227610 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.227448 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-ui-tls\") pod \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " Jul 29 08:42:18.227610 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.227484 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-offline-tls\") pod \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\" (UID: \"02d10c45-adf2-4e21-8140-f3d08b6aaa65\") " Jul 29 08:42:18.227868 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.227822 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d10c45-adf2-4e21-8140-f3d08b6aaa65-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "02d10c45-adf2-4e21-8140-f3d08b6aaa65" (UID: "02d10c45-adf2-4e21-8140-f3d08b6aaa65"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:42:18.227929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.227899 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d10c45-adf2-4e21-8140-f3d08b6aaa65-feast-data" (OuterVolumeSpecName: "feast-data") pod "02d10c45-adf2-4e21-8140-f3d08b6aaa65" (UID: "02d10c45-adf2-4e21-8140-f3d08b6aaa65"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:42:18.229929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.229902 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-online-tls" (OuterVolumeSpecName: "online-tls") pod "02d10c45-adf2-4e21-8140-f3d08b6aaa65" (UID: "02d10c45-adf2-4e21-8140-f3d08b6aaa65"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:42:18.229929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.229915 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "02d10c45-adf2-4e21-8140-f3d08b6aaa65" (UID: "02d10c45-adf2-4e21-8140-f3d08b6aaa65"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:42:18.230194 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.230145 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d10c45-adf2-4e21-8140-f3d08b6aaa65-kube-api-access-d987c" (OuterVolumeSpecName: "kube-api-access-d987c") pod "02d10c45-adf2-4e21-8140-f3d08b6aaa65" (UID: "02d10c45-adf2-4e21-8140-f3d08b6aaa65"). InnerVolumeSpecName "kube-api-access-d987c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:42:18.230194 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.230177 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "02d10c45-adf2-4e21-8140-f3d08b6aaa65" (UID: "02d10c45-adf2-4e21-8140-f3d08b6aaa65"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:42:18.328271 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.328235 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/02d10c45-adf2-4e21-8140-f3d08b6aaa65-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.328410 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.328276 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d987c\" (UniqueName: \"kubernetes.io/projected/02d10c45-adf2-4e21-8140-f3d08b6aaa65-kube-api-access-d987c\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.328410 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.328291 2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-ui-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.328410 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.328305 2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-offline-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.328410 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.328318 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02d10c45-adf2-4e21-8140-f3d08b6aaa65-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.328410 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.328329 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d10c45-adf2-4e21-8140-f3d08b6aaa65-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.396227 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.396203 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:42:18.428827 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.428796 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rmlh\" (UniqueName: \"kubernetes.io/projected/af073fc2-3744-4292-b7e2-41fac11eac8f-kube-api-access-6rmlh\") pod \"af073fc2-3744-4292-b7e2-41fac11eac8f\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " Jul 29 08:42:18.429021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.428887 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/af073fc2-3744-4292-b7e2-41fac11eac8f-feast-data\") pod \"af073fc2-3744-4292-b7e2-41fac11eac8f\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " Jul 29 08:42:18.429021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.428918 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-online-tls\") pod \"af073fc2-3744-4292-b7e2-41fac11eac8f\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " Jul 29 08:42:18.429021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.428970 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-offline-tls\") pod \"af073fc2-3744-4292-b7e2-41fac11eac8f\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " Jul 29 08:42:18.429021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.429003 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-registry-tls\") pod \"af073fc2-3744-4292-b7e2-41fac11eac8f\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " Jul 29 08:42:18.429262 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.429031 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-ui-tls\") pod \"af073fc2-3744-4292-b7e2-41fac11eac8f\" (UID: \"af073fc2-3744-4292-b7e2-41fac11eac8f\") " Jul 29 08:42:18.429550 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.429506 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af073fc2-3744-4292-b7e2-41fac11eac8f-feast-data" (OuterVolumeSpecName: "feast-data") pod "af073fc2-3744-4292-b7e2-41fac11eac8f" (UID: "af073fc2-3744-4292-b7e2-41fac11eac8f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:42:18.431447 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.431426 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "af073fc2-3744-4292-b7e2-41fac11eac8f" (UID: "af073fc2-3744-4292-b7e2-41fac11eac8f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:42:18.431768 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.431734 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "af073fc2-3744-4292-b7e2-41fac11eac8f" (UID: "af073fc2-3744-4292-b7e2-41fac11eac8f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:42:18.431768 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.431749 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-online-tls" (OuterVolumeSpecName: "online-tls") pod "af073fc2-3744-4292-b7e2-41fac11eac8f" (UID: "af073fc2-3744-4292-b7e2-41fac11eac8f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:42:18.431892 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.431837 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "af073fc2-3744-4292-b7e2-41fac11eac8f" (UID: "af073fc2-3744-4292-b7e2-41fac11eac8f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:42:18.431892 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.431835 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af073fc2-3744-4292-b7e2-41fac11eac8f-kube-api-access-6rmlh" (OuterVolumeSpecName: "kube-api-access-6rmlh") pod "af073fc2-3744-4292-b7e2-41fac11eac8f" (UID: "af073fc2-3744-4292-b7e2-41fac11eac8f"). InnerVolumeSpecName "kube-api-access-6rmlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:42:18.529959 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.529925 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/af073fc2-3744-4292-b7e2-41fac11eac8f-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.529959 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.529953 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.529959 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.529962 2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-offline-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.529959 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.529969 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.530208 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.529977 2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/af073fc2-3744-4292-b7e2-41fac11eac8f-ui-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.530208 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.529985 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rmlh\" (UniqueName: \"kubernetes.io/projected/af073fc2-3744-4292-b7e2-41fac11eac8f-kube-api-access-6rmlh\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:42:18.960361 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.960331 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" Jul 29 08:42:18.960825 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.960328 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx" event={"ID":"af073fc2-3744-4292-b7e2-41fac11eac8f","Type":"ContainerDied","Data":"810cbabbbf76e7bd4fc1447f5b5e4f9d1907c87895141e4183854fc70ae4e55a"} Jul 29 08:42:18.960825 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.960458 2582 scope.go:117] "RemoveContainer" containerID="6367967cce016f8775d3474b6a3c166adf114990406d4044d04a47e38fbcc8d7" Jul 29 08:42:18.963411 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.963385 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" Jul 29 08:42:18.963536 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.963408 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp" event={"ID":"02d10c45-adf2-4e21-8140-f3d08b6aaa65","Type":"ContainerDied","Data":"6ae39c3000a8677f16c8cf9870c77ac02c6189ca4ab1fa2f0d2c10fa2e0eedcc"} Jul 29 08:42:18.970957 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.970939 2582 scope.go:117] "RemoveContainer" containerID="c9233db724f859a386db417dce301c80ebbc0ec38e469fbf3f7af54f194cd878" Jul 29 08:42:18.978798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.978781 2582 scope.go:117] "RemoveContainer" containerID="8f89712b1279673492f30d4e8112d46d450f65754fa1ca707ca2f2a6884aabf0" Jul 29 08:42:18.985182 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.985160 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx"] Jul 29 08:42:18.986584 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.986571 2582 scope.go:117] "RemoveContainer" containerID="5751c237edf5bb4ba60ea5a05bac15e7bcd1d960de8c053682a9c3ecdd1fe8ec" Jul 29 08:42:18.989216 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.989193 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-9mpwx"] Jul 29 08:42:18.994373 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:18.994354 2582 scope.go:117] "RemoveContainer" containerID="883d2dd4e60c7df78604ba8e2cc4e4b5dc4da8b625676a5b487d8efd865ea887" Jul 29 08:42:19.002294 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:19.002275 2582 scope.go:117] "RemoveContainer" containerID="369db23630bf27323cbc92648bdd0c8b93b52c5b8f38dbf6454d2d1aacd2793c" Jul 29 08:42:19.003293 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:19.003277 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp"] Jul 29 08:42:19.007788 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:19.007767 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-wmntp"] Jul 29 08:42:19.009946 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:19.009931 2582 scope.go:117] "RemoveContainer" containerID="039d4fbb029aad335baeda8644364c2f352ac8227eab641a52026b6b838e05d0" Jul 29 08:42:19.017570 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:19.017553 2582 scope.go:117] "RemoveContainer" containerID="93b3e96f913cf1b68c475573ac28e4a4427353f222ee61618421c16eb81153d6" Jul 29 08:42:19.025072 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:19.025055 2582 scope.go:117] "RemoveContainer" containerID="9e7b2f359668ac1759040d8fd7ccc7da20c055141e250317adc721a9de6e110c" Jul 29 08:42:19.032256 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:19.032241 2582 scope.go:117] "RemoveContainer" containerID="60490df88dc99c4e5fe53bc7bcb1ee59b1f0572461e1290048611328157c41c1" Jul 29 08:42:19.039410 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:19.039394 2582 scope.go:117] "RemoveContainer" containerID="1b1629a6c0b7bf5bf146287ca4ea27d5276c2a86a05f73150431efff4fc908e5" Jul 29 08:42:20.250329 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:20.250296 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" path="/var/lib/kubelet/pods/02d10c45-adf2-4e21-8140-f3d08b6aaa65/volumes" Jul 29 08:42:20.250860 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:20.250845 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" path="/var/lib/kubelet/pods/af073fc2-3744-4292-b7e2-41fac11eac8f/volumes" Jul 29 08:42:30.980977 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:30.980932 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:31.138864 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:31.138831 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:37.230156 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:37.230115 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:37.299125 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:37.299090 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:45.814009 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:45.813975 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:58.372299 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.372260 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:58.472988 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.472955 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:58.685107 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.685014 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:58.785648 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.785601 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:58.804731 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.804693 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx"] Jul 29 08:42:58.805074 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805058 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="registry" Jul 29 08:42:58.805161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805078 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="registry" Jul 29 08:42:58.805161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805093 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="online" Jul 29 08:42:58.805161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805101 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="online" Jul 29 08:42:58.805161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805114 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="offline" Jul 29 08:42:58.805161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805122 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="offline" Jul 29 08:42:58.805161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805136 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="feast-init" Jul 29 08:42:58.805161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805144 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="feast-init" Jul 29 08:42:58.805161 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805157 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="feast-apply" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805165 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="feast-apply" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805175 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="offline" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805182 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="offline" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805193 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="feast-apply" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805201 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="feast-apply" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805210 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="online" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805218 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="online" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805235 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="feast-init" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805243 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="feast-init" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805253 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="ui" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805261 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="ui" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805284 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="ui" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805292 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="ui" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805371 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="online" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805382 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="ui" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805392 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d10c45-adf2-4e21-8140-f3d08b6aaa65" containerName="offline" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805401 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="online" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805412 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="ui" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805423 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="offline" Jul 29 08:42:58.805598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.805432 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af073fc2-3744-4292-b7e2-41fac11eac8f" containerName="registry" Jul 29 08:42:58.809374 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.809354 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.812463 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.812444 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 29 08:42:58.812589 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.812464 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 29 08:42:58.812589 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.812480 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 29 08:42:58.812768 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.812748 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 29 08:42:58.812838 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.812816 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 29 08:42:58.812909 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.812893 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-85snd\"" Jul 29 08:42:58.813080 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.813061 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 29 08:42:58.823033 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.823008 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx"] Jul 29 08:42:58.879556 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.879520 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.879556 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.879558 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkh2\" (UniqueName: \"kubernetes.io/projected/89a2df11-141b-4a32-898e-0caa2c8eeab1-kube-api-access-pmkh2\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.879834 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.879589 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.879834 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.879715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.879834 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.879754 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.879834 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.879801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89a2df11-141b-4a32-898e-0caa2c8eeab1-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.885598 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.885566 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:58.980344 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.980311 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.980529 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.980368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.980529 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.980389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.980529 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.980416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89a2df11-141b-4a32-898e-0caa2c8eeab1-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.980529 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.980443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.980773 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.980559 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkh2\" (UniqueName: \"kubernetes.io/projected/89a2df11-141b-4a32-898e-0caa2c8eeab1-kube-api-access-pmkh2\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.980773 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:42:58.980669 2582 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 29 08:42:58.980773 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:42:58.980753 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-online-tls podName:89a2df11-141b-4a32-898e-0caa2c8eeab1 nodeName:}" failed. No retries permitted until 2026-07-29 08:42:59.480735003 +0000 UTC m=+1499.793915841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-online-tls") pod "feast-simple-feast-setup-777d6cc998-mlwtx" (UID: "89a2df11-141b-4a32-898e-0caa2c8eeab1") : secret "feast-simple-feast-setup-online-tls" not found Jul 29 08:42:58.980983 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.980878 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89a2df11-141b-4a32-898e-0caa2c8eeab1-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.983198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.983169 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.983494 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.983478 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.983538 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.983490 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:58.985180 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.985160 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:58.988886 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:58.988864 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkh2\" (UniqueName: \"kubernetes.io/projected/89a2df11-141b-4a32-898e-0caa2c8eeab1-kube-api-access-pmkh2\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:59.088042 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:59.088014 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:59.187393 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:59.187361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:42:59.484971 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:59.484938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:59.487460 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:59.487440 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-mlwtx\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:59.731069 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:59.731039 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:42:59.868675 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:42:59.868651 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx"] Jul 29 08:42:59.870030 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:42:59.870004 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89a2df11_141b_4a32_898e_0caa2c8eeab1.slice/crio-c86e4de3dcb570b6c862144b50ebcb8c8bf185851e1b551ca5be11df3773c211 WatchSource:0}: Error finding container c86e4de3dcb570b6c862144b50ebcb8c8bf185851e1b551ca5be11df3773c211: Status 404 returned error can't find the container with id c86e4de3dcb570b6c862144b50ebcb8c8bf185851e1b551ca5be11df3773c211 Jul 29 08:43:00.131096 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:00.130998 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerStarted","Data":"e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd"} Jul 29 08:43:00.131096 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:00.131059 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerStarted","Data":"c86e4de3dcb570b6c862144b50ebcb8c8bf185851e1b551ca5be11df3773c211"} Jul 29 08:43:05.152744 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:05.152610 2582 generic.go:356] "Generic (PLEG): container finished" podID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerID="e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd" exitCode=0 Jul 29 08:43:05.152744 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:05.152683 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerDied","Data":"e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd"} Jul 29 08:43:06.158633 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:06.158586 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerStarted","Data":"cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2"} Jul 29 08:43:11.187067 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:11.186983 2582 generic.go:356] "Generic (PLEG): container finished" podID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerID="cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2" exitCode=0 Jul 29 08:43:11.187067 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:11.187030 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerDied","Data":"cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2"} Jul 29 08:43:12.197494 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:12.197435 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerStarted","Data":"8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52"} Jul 29 08:43:12.198014 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:12.197511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerStarted","Data":"41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8"} Jul 29 08:43:12.198014 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:12.197532 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerStarted","Data":"944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d"} Jul 29 08:43:12.198014 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:12.197549 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerStarted","Data":"11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd"} Jul 29 08:43:12.222448 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:12.222382 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podStartSLOduration=14.222360211 podStartE2EDuration="14.222360211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:43:12.220506758 +0000 UTC m=+1512.533687619" watchObservedRunningTime="2026-07-29 08:43:12.222360211 +0000 UTC m=+1512.535541071" Jul 29 08:43:14.732119 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:14.732082 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:14.732678 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:14.732654 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:14.732834 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:14.732687 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:14.732834 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:14.732705 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:14.734408 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:14.734379 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.41:6571: connect: connection refused" Jul 29 08:43:14.734640 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:14.734405 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.41:8443: connect: connection refused" Jul 29 08:43:14.734822 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:14.734406 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.41:6567/health\": dial tcp 10.134.0.41:6567: connect: connection refused" Jul 29 08:43:14.734929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:14.734441 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.41:8816: connect: connection refused" Jul 29 08:43:17.732920 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:17.732890 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:17.733463 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:17.733012 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:17.733463 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:17.733267 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:17.733685 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:17.733661 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:17.733798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:17.733700 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:17.733798 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:17.733718 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:17.733947 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:17.733922 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:17.734081 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:17.734066 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:17.734171 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:17.734159 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:17.738242 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:17.738222 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:18.232184 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:18.232154 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:18.235376 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:18.235356 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:43:20.097514 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:43:20.097411 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:44:19.907427 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:19.907326 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx"] Jul 29 08:44:19.912164 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:19.907599 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="registry" containerID="cri-o://11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd" gracePeriod=30 Jul 29 08:44:19.912164 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:19.907643 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="online" containerID="cri-o://944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d" gracePeriod=30 Jul 29 08:44:19.912164 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:19.907661 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="offline" containerID="cri-o://41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8" gracePeriod=30 Jul 29 08:44:19.912164 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:19.907684 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="ui" containerID="cri-o://8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52" gracePeriod=30 Jul 29 08:44:19.912164 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:19.910019 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:44:20.006221 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.006190 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:44:20.112838 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.112802 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:44:20.226953 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.226924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:44:20.400703 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.400672 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp"] Jul 29 08:44:20.404490 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.404471 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.410132 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.410107 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-gfzgz\"" Jul 29 08:44:20.424635 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.424596 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp"] Jul 29 08:44:20.480955 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.480869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.480955 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.480912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.480955 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.480935 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b47f876b-f8d0-497e-89fa-58b83b619157-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.481213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.480970 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.481213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.480985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.481213 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.481060 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv6k7\" (UniqueName: \"kubernetes.io/projected/b47f876b-f8d0-497e-89fa-58b83b619157-kube-api-access-wv6k7\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.488222 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.488194 2582 generic.go:356] "Generic (PLEG): container finished" podID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerID="8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52" exitCode=0 Jul 29 08:44:20.488382 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.488236 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerDied","Data":"8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52"} Jul 29 08:44:20.501033 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.501009 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:44:20.582506 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.582476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.582666 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.582533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b47f876b-f8d0-497e-89fa-58b83b619157-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.582666 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:44:20.582587 2582 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 29 08:44:20.582666 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.582595 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.582666 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.582633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.582897 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:44:20.582730 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-offline-tls podName:b47f876b-f8d0-497e-89fa-58b83b619157 nodeName:}" failed. No retries permitted until 2026-07-29 08:44:21.08265496 +0000 UTC m=+1581.395835815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-offline-tls") pod "feast-simple-feast-setup-777d6cc998-klfdp" (UID: "b47f876b-f8d0-497e-89fa-58b83b619157") : secret "feast-simple-feast-setup-offline-tls" not found Jul 29 08:44:20.582897 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.582806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv6k7\" (UniqueName: \"kubernetes.io/projected/b47f876b-f8d0-497e-89fa-58b83b619157-kube-api-access-wv6k7\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.582897 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.582861 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.583058 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.582995 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b47f876b-f8d0-497e-89fa-58b83b619157-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.585661 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.585611 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.585784 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.585611 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.585784 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.585679 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.587185 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.587133 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:44:20.591355 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.591332 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv6k7\" (UniqueName: \"kubernetes.io/projected/b47f876b-f8d0-497e-89fa-58b83b619157-kube-api-access-wv6k7\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:20.686987 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.686963 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:44:20.974377 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:20.974346 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:44:21.087741 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:21.087695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:21.090174 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:21.090155 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-klfdp\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:21.313389 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:21.313294 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:21.451201 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:21.451107 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp"] Jul 29 08:44:21.453685 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:44:21.453657 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb47f876b_f8d0_497e_89fa_58b83b619157.slice/crio-2690f7e3d0fb44fc0e7ca85dcedf0279cac704acf4eaf9010fba5974aee3f6e9 WatchSource:0}: Error finding container 2690f7e3d0fb44fc0e7ca85dcedf0279cac704acf4eaf9010fba5974aee3f6e9: Status 404 returned error can't find the container with id 2690f7e3d0fb44fc0e7ca85dcedf0279cac704acf4eaf9010fba5974aee3f6e9 Jul 29 08:44:21.493113 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:21.493080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerStarted","Data":"2690f7e3d0fb44fc0e7ca85dcedf0279cac704acf4eaf9010fba5974aee3f6e9"} Jul 29 08:44:21.496418 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:21.496388 2582 generic.go:356] "Generic (PLEG): container finished" podID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerID="944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d" exitCode=0 Jul 29 08:44:21.496528 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:21.496473 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerDied","Data":"944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d"} Jul 29 08:44:22.502985 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:22.502947 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerStarted","Data":"a6ed2e0fe91cb9771cb999e9a56a5bc79520fa4ea60439c9c60b2e9cd900a2cf"} Jul 29 08:44:26.519469 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:26.519434 2582 generic.go:356] "Generic (PLEG): container finished" podID="b47f876b-f8d0-497e-89fa-58b83b619157" containerID="a6ed2e0fe91cb9771cb999e9a56a5bc79520fa4ea60439c9c60b2e9cd900a2cf" exitCode=0 Jul 29 08:44:26.519469 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:26.519473 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerDied","Data":"a6ed2e0fe91cb9771cb999e9a56a5bc79520fa4ea60439c9c60b2e9cd900a2cf"} Jul 29 08:44:27.525700 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:27.525663 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerStarted","Data":"7e1221619acf59c83ccec30dec6de83a4ee85a8b2da16785697cb1799c827cda"} Jul 29 08:44:27.734183 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:27.734126 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.41:8443: connect: connection refused" Jul 29 08:44:28.232810 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:28.232765 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.41:6567/health\": dial tcp 10.134.0.41:6567: connect: connection refused" Jul 29 08:44:32.550927 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:32.550895 2582 generic.go:356] "Generic (PLEG): container finished" podID="b47f876b-f8d0-497e-89fa-58b83b619157" containerID="7e1221619acf59c83ccec30dec6de83a4ee85a8b2da16785697cb1799c827cda" exitCode=0 Jul 29 08:44:32.551279 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:32.550938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerDied","Data":"7e1221619acf59c83ccec30dec6de83a4ee85a8b2da16785697cb1799c827cda"} Jul 29 08:44:33.561869 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:33.561779 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerStarted","Data":"cb013d8603076a07e06ceac8d82de6534619d617532283de60cd9c439b0e1869"} Jul 29 08:44:33.561869 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:33.561856 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerStarted","Data":"721bda34f1bf35e7c13c04cb98eab5b67e3f01cfc31816549b9e0c424525d306"} Jul 29 08:44:33.562444 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:33.561880 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerStarted","Data":"949b8871071d85eaabe0c00dc4ec8cd409c5cf343e17bc76d6107f3b4ee79556"} Jul 29 08:44:33.562444 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:33.561899 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerStarted","Data":"6ac19838c716a9755bf78d9e4651aa7810373c5aff3f020e3018f6d5d83f221e"} Jul 29 08:44:33.592154 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:33.592089 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podStartSLOduration=13.592067058 podStartE2EDuration="13.592067058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:44:33.588957202 +0000 UTC m=+1593.902138079" watchObservedRunningTime="2026-07-29 08:44:33.592067058 +0000 UTC m=+1593.905247921" Jul 29 08:44:36.314414 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:36.314377 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:36.315030 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:36.314999 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:36.315030 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:36.315030 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:36.315230 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:36.315047 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:36.316718 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:36.316679 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.42:8443: connect: connection refused" Jul 29 08:44:36.316838 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:36.316687 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.42:6571: connect: connection refused" Jul 29 08:44:36.316838 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:36.316694 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="online" probeResult="failure" output="Get \"https://10.134.0.42:6567/health\": dial tcp 10.134.0.42:6567: connect: connection refused" Jul 29 08:44:36.316985 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:36.316957 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.42:8816: connect: connection refused" Jul 29 08:44:37.734026 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:37.733972 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.41:8443: connect: connection refused" Jul 29 08:44:38.232726 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:38.232677 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.41:6567/health\": dial tcp 10.134.0.41:6567: connect: connection refused" Jul 29 08:44:39.314426 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.314397 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.314856 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.314567 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.315098 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.315067 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.315262 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.315133 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.315326 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.315273 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.315584 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.315568 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.315807 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.315795 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.319746 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.319725 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.603757 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.603663 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.603757 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.603716 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.604194 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.604178 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:39.606929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:39.606913 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:44:47.734465 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:47.734420 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.41:8443: connect: connection refused" Jul 29 08:44:47.734990 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:47.734565 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:44:48.233053 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:48.233012 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.41:6567/health\": dial tcp 10.134.0.41:6567: connect: connection refused" Jul 29 08:44:48.233239 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:48.233135 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:44:50.561806 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.561779 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:44:50.650648 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.650603 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-offline-tls\") pod \"89a2df11-141b-4a32-898e-0caa2c8eeab1\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " Jul 29 08:44:50.650955 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.650688 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-ui-tls\") pod \"89a2df11-141b-4a32-898e-0caa2c8eeab1\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " Jul 29 08:44:50.650955 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.650733 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-registry-tls\") pod \"89a2df11-141b-4a32-898e-0caa2c8eeab1\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " Jul 29 08:44:50.650955 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.650790 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmkh2\" (UniqueName: \"kubernetes.io/projected/89a2df11-141b-4a32-898e-0caa2c8eeab1-kube-api-access-pmkh2\") pod \"89a2df11-141b-4a32-898e-0caa2c8eeab1\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " Jul 29 08:44:50.650955 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.650849 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89a2df11-141b-4a32-898e-0caa2c8eeab1-feast-data\") pod \"89a2df11-141b-4a32-898e-0caa2c8eeab1\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " Jul 29 08:44:50.650955 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.650880 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-online-tls\") pod \"89a2df11-141b-4a32-898e-0caa2c8eeab1\" (UID: \"89a2df11-141b-4a32-898e-0caa2c8eeab1\") " Jul 29 08:44:50.652026 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.651601 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a2df11-141b-4a32-898e-0caa2c8eeab1-feast-data" (OuterVolumeSpecName: "feast-data") pod "89a2df11-141b-4a32-898e-0caa2c8eeab1" (UID: "89a2df11-141b-4a32-898e-0caa2c8eeab1"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:44:50.653192 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653158 2582 generic.go:356] "Generic (PLEG): container finished" podID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerID="41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8" exitCode=137 Jul 29 08:44:50.653192 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653189 2582 generic.go:356] "Generic (PLEG): container finished" podID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerID="11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd" exitCode=137 Jul 29 08:44:50.653369 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653292 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerDied","Data":"41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8"} Jul 29 08:44:50.653369 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653315 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" Jul 29 08:44:50.653369 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653343 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerDied","Data":"11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd"} Jul 29 08:44:50.653369 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653365 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx" event={"ID":"89a2df11-141b-4a32-898e-0caa2c8eeab1","Type":"ContainerDied","Data":"c86e4de3dcb570b6c862144b50ebcb8c8bf185851e1b551ca5be11df3773c211"} Jul 29 08:44:50.653562 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653392 2582 scope.go:117] "RemoveContainer" containerID="8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52" Jul 29 08:44:50.653562 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653420 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "89a2df11-141b-4a32-898e-0caa2c8eeab1" (UID: "89a2df11-141b-4a32-898e-0caa2c8eeab1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:44:50.653562 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653459 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "89a2df11-141b-4a32-898e-0caa2c8eeab1" (UID: "89a2df11-141b-4a32-898e-0caa2c8eeab1"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:44:50.653735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653600 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "89a2df11-141b-4a32-898e-0caa2c8eeab1" (UID: "89a2df11-141b-4a32-898e-0caa2c8eeab1"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:44:50.653787 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.653735 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a2df11-141b-4a32-898e-0caa2c8eeab1-kube-api-access-pmkh2" (OuterVolumeSpecName: "kube-api-access-pmkh2") pod "89a2df11-141b-4a32-898e-0caa2c8eeab1" (UID: "89a2df11-141b-4a32-898e-0caa2c8eeab1"). InnerVolumeSpecName "kube-api-access-pmkh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:44:50.654261 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.654240 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-online-tls" (OuterVolumeSpecName: "online-tls") pod "89a2df11-141b-4a32-898e-0caa2c8eeab1" (UID: "89a2df11-141b-4a32-898e-0caa2c8eeab1"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:44:50.666725 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.666706 2582 scope.go:117] "RemoveContainer" containerID="41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8" Jul 29 08:44:50.674364 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.674345 2582 scope.go:117] "RemoveContainer" containerID="944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d" Jul 29 08:44:50.681895 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.681873 2582 scope.go:117] "RemoveContainer" containerID="11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd" Jul 29 08:44:50.689247 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.689227 2582 scope.go:117] "RemoveContainer" containerID="cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2" Jul 29 08:44:50.698600 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.698578 2582 scope.go:117] "RemoveContainer" containerID="e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd" Jul 29 08:44:50.705880 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.705862 2582 scope.go:117] "RemoveContainer" containerID="8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52" Jul 29 08:44:50.706142 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:44:50.706119 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52\": container with ID starting with 8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52 not found: ID does not exist" containerID="8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52" Jul 29 08:44:50.706219 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.706147 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52"} err="failed to get container status \"8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52\": rpc error: code = NotFound desc = could not find container \"8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52\": container with ID starting with 8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52 not found: ID does not exist" Jul 29 08:44:50.706219 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.706166 2582 scope.go:117] "RemoveContainer" containerID="41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8" Jul 29 08:44:50.706402 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:44:50.706385 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8\": container with ID starting with 41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8 not found: ID does not exist" containerID="41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8" Jul 29 08:44:50.706442 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.706413 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8"} err="failed to get container status \"41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8\": rpc error: code = NotFound desc = could not find container \"41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8\": container with ID starting with 41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8 not found: ID does not exist" Jul 29 08:44:50.706442 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.706430 2582 scope.go:117] "RemoveContainer" containerID="944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d" Jul 29 08:44:50.706671 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:44:50.706649 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d\": container with ID starting with 944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d not found: ID does not exist" containerID="944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d" Jul 29 08:44:50.706735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.706679 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d"} err="failed to get container status \"944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d\": rpc error: code = NotFound desc = could not find container \"944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d\": container with ID starting with 944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d not found: ID does not exist" Jul 29 08:44:50.706735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.706699 2582 scope.go:117] "RemoveContainer" containerID="11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd" Jul 29 08:44:50.706908 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:44:50.706895 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd\": container with ID starting with 11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd not found: ID does not exist" containerID="11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd" Jul 29 08:44:50.706948 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.706912 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd"} err="failed to get container status \"11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd\": rpc error: code = NotFound desc = could not find container \"11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd\": container with ID starting with 11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd not found: ID does not exist" Jul 29 08:44:50.706948 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.706925 2582 scope.go:117] "RemoveContainer" containerID="cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2" Jul 29 08:44:50.707094 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:44:50.707079 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2\": container with ID starting with cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2 not found: ID does not exist" containerID="cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2" Jul 29 08:44:50.707140 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.707098 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2"} err="failed to get container status \"cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2\": rpc error: code = NotFound desc = could not find container \"cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2\": container with ID starting with cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2 not found: ID does not exist" Jul 29 08:44:50.707140 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.707111 2582 scope.go:117] "RemoveContainer" containerID="e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd" Jul 29 08:44:50.707298 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:44:50.707283 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd\": container with ID starting with e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd not found: ID does not exist" containerID="e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd" Jul 29 08:44:50.707341 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.707301 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd"} err="failed to get container status \"e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd\": rpc error: code = NotFound desc = could not find container \"e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd\": container with ID starting with e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd not found: ID does not exist" Jul 29 08:44:50.707341 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.707311 2582 scope.go:117] "RemoveContainer" containerID="8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52" Jul 29 08:44:50.707483 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.707466 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52"} err="failed to get container status \"8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52\": rpc error: code = NotFound desc = could not find container \"8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52\": container with ID starting with 8b3339a3a6c174f28dfcc33c81a6e185c2c4861b12ac8b742cccd859d1d77c52 not found: ID does not exist" Jul 29 08:44:50.707533 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.707485 2582 scope.go:117] "RemoveContainer" containerID="41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8" Jul 29 08:44:50.707735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.707719 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8"} err="failed to get container status \"41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8\": rpc error: code = NotFound desc = could not find container \"41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8\": container with ID starting with 41a17d8aa092bf12f22eb8d923e5c0c0f08a3cf261fc06ac3efcb632d5dd8ae8 not found: ID does not exist" Jul 29 08:44:50.707735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.707735 2582 scope.go:117] "RemoveContainer" containerID="944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d" Jul 29 08:44:50.707971 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.707952 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d"} err="failed to get container status \"944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d\": rpc error: code = NotFound desc = could not find container \"944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d\": container with ID starting with 944133657cffefb7115fc488c245daf8a187321a2397ed81df5f1ac00637228d not found: ID does not exist" Jul 29 08:44:50.708021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.707973 2582 scope.go:117] "RemoveContainer" containerID="11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd" Jul 29 08:44:50.708173 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.708158 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd"} err="failed to get container status \"11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd\": rpc error: code = NotFound desc = could not find container \"11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd\": container with ID starting with 11825fcd1a8ac992c4e91661364c408611bdf7e9e14cafb117c1d226efb927cd not found: ID does not exist" Jul 29 08:44:50.708215 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.708172 2582 scope.go:117] "RemoveContainer" containerID="cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2" Jul 29 08:44:50.708333 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.708316 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2"} err="failed to get container status \"cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2\": rpc error: code = NotFound desc = could not find container \"cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2\": container with ID starting with cf41d5f15f196632b50616b41bba62bb64ac8e7dd8760e5be8d55b83e72b38b2 not found: ID does not exist" Jul 29 08:44:50.708374 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.708334 2582 scope.go:117] "RemoveContainer" containerID="e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd" Jul 29 08:44:50.708541 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.708524 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd"} err="failed to get container status \"e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd\": rpc error: code = NotFound desc = could not find container \"e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd\": container with ID starting with e7a77e89e0e557816d8c13f5b5a5a95014dcea4142075543f5178e72df301ccd not found: ID does not exist" Jul 29 08:44:50.751747 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.751715 2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-offline-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:44:50.751747 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.751741 2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-ui-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:44:50.751747 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.751749 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:44:50.751974 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.751758 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmkh2\" (UniqueName: \"kubernetes.io/projected/89a2df11-141b-4a32-898e-0caa2c8eeab1-kube-api-access-pmkh2\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:44:50.751974 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.751768 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89a2df11-141b-4a32-898e-0caa2c8eeab1-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:44:50.751974 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.751778 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89a2df11-141b-4a32-898e-0caa2c8eeab1-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:44:50.975374 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.975345 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx"] Jul 29 08:44:50.979103 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:50.979080 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-mlwtx"] Jul 29 08:44:52.251298 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:44:52.251261 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" path="/var/lib/kubelet/pods/89a2df11-141b-4a32-898e-0caa2c8eeab1/volumes" Jul 29 08:45:41.347829 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.347793 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:45:41.676083 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.675995 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn"] Jul 29 08:45:41.676414 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676396 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="offline" Jul 29 08:45:41.676500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676416 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="offline" Jul 29 08:45:41.676500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676448 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="online" Jul 29 08:45:41.676500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676457 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="online" Jul 29 08:45:41.676500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676469 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="ui" Jul 29 08:45:41.676500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676478 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="ui" Jul 29 08:45:41.676500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676492 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="feast-init" Jul 29 08:45:41.676500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676500 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="feast-init" Jul 29 08:45:41.677020 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676513 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="registry" Jul 29 08:45:41.677020 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676522 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="registry" Jul 29 08:45:41.677020 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676536 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="feast-apply" Jul 29 08:45:41.677020 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676545 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="feast-apply" Jul 29 08:45:41.677020 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676649 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="online" Jul 29 08:45:41.677020 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676664 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="offline" Jul 29 08:45:41.677020 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676674 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="ui" Jul 29 08:45:41.677020 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.676688 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a2df11-141b-4a32-898e-0caa2c8eeab1" containerName="registry" Jul 29 08:45:41.680412 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.680391 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.683067 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.683044 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 29 08:45:41.683190 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.683051 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 29 08:45:41.683190 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.683119 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 29 08:45:41.683190 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.683086 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 29 08:45:41.683190 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.683124 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-vvmfx\"" Jul 29 08:45:41.683413 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.683262 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 29 08:45:41.683413 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.683302 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 29 08:45:41.693333 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.693312 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn"] Jul 29 08:45:41.781286 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.781254 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d86643-8041-470c-b991-d3d51d2790a9-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.781455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.781298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.781455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.781324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.781455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.781350 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.781455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.781381 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e7d86643-8041-470c-b991-d3d51d2790a9-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.781455 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.781397 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6tz\" (UniqueName: \"kubernetes.io/projected/e7d86643-8041-470c-b991-d3d51d2790a9-kube-api-access-8t6tz\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.881973 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.881944 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.881973 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.881979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.882253 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.881999 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.882253 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.882034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e7d86643-8041-470c-b991-d3d51d2790a9-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.882253 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.882061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6tz\" (UniqueName: \"kubernetes.io/projected/e7d86643-8041-470c-b991-d3d51d2790a9-kube-api-access-8t6tz\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.882253 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:45:41.882092 2582 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 29 08:45:41.882253 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:45:41.882117 2582 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jul 29 08:45:41.882253 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.882143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d86643-8041-470c-b991-d3d51d2790a9-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.882253 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:45:41.882170 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-offline-tls podName:e7d86643-8041-470c-b991-d3d51d2790a9 nodeName:}" failed. No retries permitted until 2026-07-29 08:45:42.382147075 +0000 UTC m=+1662.695327914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-offline-tls") pod "feast-simple-feast-remote-setup-db6955c6c-2hvfn" (UID: "e7d86643-8041-470c-b991-d3d51d2790a9") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 29 08:45:41.882253 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:45:41.882244 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-online-tls podName:e7d86643-8041-470c-b991-d3d51d2790a9 nodeName:}" failed. No retries permitted until 2026-07-29 08:45:42.382209894 +0000 UTC m=+1662.695390749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-online-tls") pod "feast-simple-feast-remote-setup-db6955c6c-2hvfn" (UID: "e7d86643-8041-470c-b991-d3d51d2790a9") : secret "feast-simple-feast-remote-setup-online-tls" not found Jul 29 08:45:41.882589 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.882559 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d86643-8041-470c-b991-d3d51d2790a9-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.882717 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.882699 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e7d86643-8041-470c-b991-d3d51d2790a9-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.884635 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.884599 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:41.893846 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:41.893825 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6tz\" (UniqueName: \"kubernetes.io/projected/e7d86643-8041-470c-b991-d3d51d2790a9-kube-api-access-8t6tz\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:42.386690 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:42.386656 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:42.386690 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:42.386693 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:42.389198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:42.389174 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:42.389318 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:42.389216 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-2hvfn\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:42.589783 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:42.589747 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:42.725105 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:42.725070 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn"] Jul 29 08:45:42.728363 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:45:42.728337 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7d86643_8041_470c_b991_d3d51d2790a9.slice/crio-7dd252aacafbb7ce57a3620eeee325b0c3bade732fab76ba69e7e5b6ac5b3642 WatchSource:0}: Error finding container 7dd252aacafbb7ce57a3620eeee325b0c3bade732fab76ba69e7e5b6ac5b3642: Status 404 returned error can't find the container with id 7dd252aacafbb7ce57a3620eeee325b0c3bade732fab76ba69e7e5b6ac5b3642 Jul 29 08:45:42.868242 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:42.868207 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerStarted","Data":"d5bceac6f46ca6a815bd6f41fb61ea0b8b096fe40501f6acf5ca00120a74125e"} Jul 29 08:45:42.868242 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:42.868254 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerStarted","Data":"7dd252aacafbb7ce57a3620eeee325b0c3bade732fab76ba69e7e5b6ac5b3642"} Jul 29 08:45:43.692274 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:43.692239 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:45:47.893018 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:47.892930 2582 generic.go:356] "Generic (PLEG): container finished" podID="e7d86643-8041-470c-b991-d3d51d2790a9" containerID="d5bceac6f46ca6a815bd6f41fb61ea0b8b096fe40501f6acf5ca00120a74125e" exitCode=0 Jul 29 08:45:47.893018 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:47.893003 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerDied","Data":"d5bceac6f46ca6a815bd6f41fb61ea0b8b096fe40501f6acf5ca00120a74125e"} Jul 29 08:45:48.899709 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:48.899672 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerStarted","Data":"219c55c3cab4fad4702a461c322feeca9fb5692909c3c4d1d83fce449af8745e"} Jul 29 08:45:53.925080 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:53.925048 2582 generic.go:356] "Generic (PLEG): container finished" podID="e7d86643-8041-470c-b991-d3d51d2790a9" containerID="219c55c3cab4fad4702a461c322feeca9fb5692909c3c4d1d83fce449af8745e" exitCode=0 Jul 29 08:45:53.925473 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:53.925125 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerDied","Data":"219c55c3cab4fad4702a461c322feeca9fb5692909c3c4d1d83fce449af8745e"} Jul 29 08:45:54.934462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:54.934421 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerStarted","Data":"9c5d48007db3eea9f41e86743c5c6408841381093edb858199d5bd393eeda72e"} Jul 29 08:45:54.934966 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:54.934487 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerStarted","Data":"b634fa4280525edafed5989b9c234ad2a183a8df5acf71fbff98d75c8006467b"} Jul 29 08:45:54.934966 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:54.934509 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerStarted","Data":"d083839a23dddb3b9d51ae49dc6ab32d8e54f6d58c0dcc87edc4545efd25915e"} Jul 29 08:45:54.962435 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:54.962359 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podStartSLOduration=13.962335656 podStartE2EDuration="13.962335656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:45:54.960406551 +0000 UTC m=+1675.273587415" watchObservedRunningTime="2026-07-29 08:45:54.962335656 +0000 UTC m=+1675.275516518" Jul 29 08:45:57.590140 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:57.590098 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:57.590750 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:57.590727 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:57.590903 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:57.590760 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:45:57.592023 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:57.591992 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.43:6567/health\": dial tcp 10.134.0.43:6567: connect: connection refused" Jul 29 08:45:57.592185 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:57.591987 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.43:8443: connect: connection refused" Jul 29 08:45:57.592185 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:45:57.591987 2582 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.43:8816: connect: connection refused" Jul 29 08:46:00.590898 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:46:00.590864 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:46:00.591499 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:46:00.591470 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:46:00.591573 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:46:00.591520 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:46:00.591718 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:46:00.591700 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:46:00.597656 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:46:00.597633 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:46:00.964533 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:46:00.964503 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:46:00.964860 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:46:00.964546 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:46:00.965142 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:46:00.965124 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:46:00.969273 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:46:00.969250 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:47:02.754003 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.753963 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:02.755314 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.755287 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn"] Jul 29 08:47:02.755560 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.755533 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="online" containerID="cri-o://d083839a23dddb3b9d51ae49dc6ab32d8e54f6d58c0dcc87edc4545efd25915e" gracePeriod=30 Jul 29 08:47:02.755676 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.755583 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="ui" containerID="cri-o://9c5d48007db3eea9f41e86743c5c6408841381093edb858199d5bd393eeda72e" gracePeriod=30 Jul 29 08:47:02.755746 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.755708 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="offline" containerID="cri-o://b634fa4280525edafed5989b9c234ad2a183a8df5acf71fbff98d75c8006467b" gracePeriod=30 Jul 29 08:47:02.873876 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.873844 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:02.970208 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.970170 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp"] Jul 29 08:47:02.970676 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.970460 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="registry" containerID="cri-o://6ac19838c716a9755bf78d9e4651aa7810373c5aff3f020e3018f6d5d83f221e" gracePeriod=30 Jul 29 08:47:02.970676 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.970528 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="online" containerID="cri-o://949b8871071d85eaabe0c00dc4ec8cd409c5cf343e17bc76d6107f3b4ee79556" gracePeriod=30 Jul 29 08:47:02.970676 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.970532 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="ui" containerID="cri-o://cb013d8603076a07e06ceac8d82de6534619d617532283de60cd9c439b0e1869" gracePeriod=30 Jul 29 08:47:02.970676 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.970555 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="offline" containerID="cri-o://721bda34f1bf35e7c13c04cb98eab5b67e3f01cfc31816549b9e0c424525d306" gracePeriod=30 Jul 29 08:47:02.973924 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:02.973896 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:03.039068 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:03.038970 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:04.230859 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:04.230820 2582 generic.go:356] "Generic (PLEG): container finished" podID="e7d86643-8041-470c-b991-d3d51d2790a9" containerID="9c5d48007db3eea9f41e86743c5c6408841381093edb858199d5bd393eeda72e" exitCode=0 Jul 29 08:47:04.230859 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:04.230847 2582 generic.go:356] "Generic (PLEG): container finished" podID="e7d86643-8041-470c-b991-d3d51d2790a9" containerID="d083839a23dddb3b9d51ae49dc6ab32d8e54f6d58c0dcc87edc4545efd25915e" exitCode=0 Jul 29 08:47:04.231313 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:04.230897 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerDied","Data":"9c5d48007db3eea9f41e86743c5c6408841381093edb858199d5bd393eeda72e"} Jul 29 08:47:04.231313 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:04.230937 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerDied","Data":"d083839a23dddb3b9d51ae49dc6ab32d8e54f6d58c0dcc87edc4545efd25915e"} Jul 29 08:47:04.233722 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:04.233698 2582 generic.go:356] "Generic (PLEG): container finished" podID="b47f876b-f8d0-497e-89fa-58b83b619157" containerID="cb013d8603076a07e06ceac8d82de6534619d617532283de60cd9c439b0e1869" exitCode=0 Jul 29 08:47:04.233722 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:04.233716 2582 generic.go:356] "Generic (PLEG): container finished" podID="b47f876b-f8d0-497e-89fa-58b83b619157" containerID="949b8871071d85eaabe0c00dc4ec8cd409c5cf343e17bc76d6107f3b4ee79556" exitCode=0 Jul 29 08:47:04.233910 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:04.233739 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerDied","Data":"cb013d8603076a07e06ceac8d82de6534619d617532283de60cd9c439b0e1869"} Jul 29 08:47:04.233910 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:04.233772 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerDied","Data":"949b8871071d85eaabe0c00dc4ec8cd409c5cf343e17bc76d6107f3b4ee79556"} Jul 29 08:47:08.195372 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:08.195335 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:09.316135 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:09.316097 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.42:8443: connect: connection refused" Jul 29 08:47:09.604393 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:09.604295 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="online" probeResult="failure" output="Get \"https://10.134.0.42:6567/health\": dial tcp 10.134.0.42:6567: connect: connection refused" Jul 29 08:47:10.965489 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:10.965438 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.43:8443: connect: connection refused" Jul 29 08:47:10.965906 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:10.965488 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.43:6567/health\": dial tcp 10.134.0.43:6567: connect: connection refused" Jul 29 08:47:14.826418 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:14.826384 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:19.316148 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:19.316097 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.42:8443: connect: connection refused" Jul 29 08:47:19.604727 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:19.604601 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="online" probeResult="failure" output="Get \"https://10.134.0.42:6567/health\": dial tcp 10.134.0.42:6567: connect: connection refused" Jul 29 08:47:20.965362 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:20.965321 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.43:8443: connect: connection refused" Jul 29 08:47:20.965792 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:20.965382 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.43:6567/health\": dial tcp 10.134.0.43:6567: connect: connection refused" Jul 29 08:47:29.315929 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:29.315889 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.42:8443: connect: connection refused" Jul 29 08:47:29.316358 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:29.316034 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:47:29.604864 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:29.604763 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="online" probeResult="failure" output="Get \"https://10.134.0.42:6567/health\": dial tcp 10.134.0.42:6567: connect: connection refused" Jul 29 08:47:29.605058 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:29.604894 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:47:30.965328 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:30.965282 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.43:8443: connect: connection refused" Jul 29 08:47:30.965787 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:30.965300 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.43:6567/health\": dial tcp 10.134.0.43:6567: connect: connection refused" Jul 29 08:47:30.965787 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:30.965415 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:47:30.965787 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:30.965483 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:47:33.355802 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.355758 2582 generic.go:356] "Generic (PLEG): container finished" podID="b47f876b-f8d0-497e-89fa-58b83b619157" containerID="721bda34f1bf35e7c13c04cb98eab5b67e3f01cfc31816549b9e0c424525d306" exitCode=137 Jul 29 08:47:33.355802 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.355791 2582 generic.go:356] "Generic (PLEG): container finished" podID="b47f876b-f8d0-497e-89fa-58b83b619157" containerID="6ac19838c716a9755bf78d9e4651aa7810373c5aff3f020e3018f6d5d83f221e" exitCode=137 Jul 29 08:47:33.356303 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.355821 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerDied","Data":"721bda34f1bf35e7c13c04cb98eab5b67e3f01cfc31816549b9e0c424525d306"} Jul 29 08:47:33.356303 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.355867 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerDied","Data":"6ac19838c716a9755bf78d9e4651aa7810373c5aff3f020e3018f6d5d83f221e"} Jul 29 08:47:33.358823 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.358797 2582 generic.go:356] "Generic (PLEG): container finished" podID="e7d86643-8041-470c-b991-d3d51d2790a9" containerID="b634fa4280525edafed5989b9c234ad2a183a8df5acf71fbff98d75c8006467b" exitCode=137 Jul 29 08:47:33.358953 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.358871 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerDied","Data":"b634fa4280525edafed5989b9c234ad2a183a8df5acf71fbff98d75c8006467b"} Jul 29 08:47:33.417549 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.417522 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:47:33.576918 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.576886 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-offline-tls\") pod \"e7d86643-8041-470c-b991-d3d51d2790a9\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " Jul 29 08:47:33.577139 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.576931 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-ui-tls\") pod \"e7d86643-8041-470c-b991-d3d51d2790a9\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " Jul 29 08:47:33.577139 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.576975 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t6tz\" (UniqueName: \"kubernetes.io/projected/e7d86643-8041-470c-b991-d3d51d2790a9-kube-api-access-8t6tz\") pod \"e7d86643-8041-470c-b991-d3d51d2790a9\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " Jul 29 08:47:33.577139 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.577002 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-online-tls\") pod \"e7d86643-8041-470c-b991-d3d51d2790a9\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " Jul 29 08:47:33.577139 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.577100 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d86643-8041-470c-b991-d3d51d2790a9-feast-data\") pod \"e7d86643-8041-470c-b991-d3d51d2790a9\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " Jul 29 08:47:33.577139 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.577127 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e7d86643-8041-470c-b991-d3d51d2790a9-registry-tls\") pod \"e7d86643-8041-470c-b991-d3d51d2790a9\" (UID: \"e7d86643-8041-470c-b991-d3d51d2790a9\") " Jul 29 08:47:33.577711 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.577674 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d86643-8041-470c-b991-d3d51d2790a9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e7d86643-8041-470c-b991-d3d51d2790a9" (UID: "e7d86643-8041-470c-b991-d3d51d2790a9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:47:33.578011 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.577898 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d86643-8041-470c-b991-d3d51d2790a9-feast-data" (OuterVolumeSpecName: "feast-data") pod "e7d86643-8041-470c-b991-d3d51d2790a9" (UID: "e7d86643-8041-470c-b991-d3d51d2790a9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:47:33.579590 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.579564 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e7d86643-8041-470c-b991-d3d51d2790a9" (UID: "e7d86643-8041-470c-b991-d3d51d2790a9"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:33.579590 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.579574 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e7d86643-8041-470c-b991-d3d51d2790a9" (UID: "e7d86643-8041-470c-b991-d3d51d2790a9"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:33.579748 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.579568 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d86643-8041-470c-b991-d3d51d2790a9-kube-api-access-8t6tz" (OuterVolumeSpecName: "kube-api-access-8t6tz") pod "e7d86643-8041-470c-b991-d3d51d2790a9" (UID: "e7d86643-8041-470c-b991-d3d51d2790a9"). InnerVolumeSpecName "kube-api-access-8t6tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:47:33.579910 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.579891 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-online-tls" (OuterVolumeSpecName: "online-tls") pod "e7d86643-8041-470c-b991-d3d51d2790a9" (UID: "e7d86643-8041-470c-b991-d3d51d2790a9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:33.622026 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.622000 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:47:33.678193 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.678161 2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-offline-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.678193 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.678188 2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-ui-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.678193 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.678197 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t6tz\" (UniqueName: \"kubernetes.io/projected/e7d86643-8041-470c-b991-d3d51d2790a9-kube-api-access-8t6tz\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.678431 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.678207 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e7d86643-8041-470c-b991-d3d51d2790a9-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.678431 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.678215 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d86643-8041-470c-b991-d3d51d2790a9-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.678431 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.678222 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e7d86643-8041-470c-b991-d3d51d2790a9-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.779319 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.779283 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-offline-tls\") pod \"b47f876b-f8d0-497e-89fa-58b83b619157\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " Jul 29 08:47:33.779319 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.779322 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b47f876b-f8d0-497e-89fa-58b83b619157-feast-data\") pod \"b47f876b-f8d0-497e-89fa-58b83b619157\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " Jul 29 08:47:33.779535 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.779344 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv6k7\" (UniqueName: \"kubernetes.io/projected/b47f876b-f8d0-497e-89fa-58b83b619157-kube-api-access-wv6k7\") pod \"b47f876b-f8d0-497e-89fa-58b83b619157\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " Jul 29 08:47:33.779535 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.779386 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-registry-tls\") pod \"b47f876b-f8d0-497e-89fa-58b83b619157\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " Jul 29 08:47:33.779535 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.779503 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-ui-tls\") pod \"b47f876b-f8d0-497e-89fa-58b83b619157\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " Jul 29 08:47:33.779682 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.779574 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-online-tls\") pod \"b47f876b-f8d0-497e-89fa-58b83b619157\" (UID: \"b47f876b-f8d0-497e-89fa-58b83b619157\") " Jul 29 08:47:33.779985 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.779952 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47f876b-f8d0-497e-89fa-58b83b619157-feast-data" (OuterVolumeSpecName: "feast-data") pod "b47f876b-f8d0-497e-89fa-58b83b619157" (UID: "b47f876b-f8d0-497e-89fa-58b83b619157"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:47:33.781841 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.781817 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47f876b-f8d0-497e-89fa-58b83b619157-kube-api-access-wv6k7" (OuterVolumeSpecName: "kube-api-access-wv6k7") pod "b47f876b-f8d0-497e-89fa-58b83b619157" (UID: "b47f876b-f8d0-497e-89fa-58b83b619157"). InnerVolumeSpecName "kube-api-access-wv6k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:47:33.781841 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.781827 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b47f876b-f8d0-497e-89fa-58b83b619157" (UID: "b47f876b-f8d0-497e-89fa-58b83b619157"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:33.782000 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.781916 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "b47f876b-f8d0-497e-89fa-58b83b619157" (UID: "b47f876b-f8d0-497e-89fa-58b83b619157"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:33.782156 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.782140 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b47f876b-f8d0-497e-89fa-58b83b619157" (UID: "b47f876b-f8d0-497e-89fa-58b83b619157"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:33.782208 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.782156 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-online-tls" (OuterVolumeSpecName: "online-tls") pod "b47f876b-f8d0-497e-89fa-58b83b619157" (UID: "b47f876b-f8d0-497e-89fa-58b83b619157"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:33.880554 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.880471 2582 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-offline-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.880554 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.880496 2582 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b47f876b-f8d0-497e-89fa-58b83b619157-feast-data\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.880554 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.880505 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv6k7\" (UniqueName: \"kubernetes.io/projected/b47f876b-f8d0-497e-89fa-58b83b619157-kube-api-access-wv6k7\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.880554 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.880515 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-registry-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.880554 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.880524 2582 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-ui-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:33.880554 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:33.880531 2582 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b47f876b-f8d0-497e-89fa-58b83b619157-online-tls\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:47:34.365948 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.365924 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" Jul 29 08:47:34.365948 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.365925 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp" event={"ID":"b47f876b-f8d0-497e-89fa-58b83b619157","Type":"ContainerDied","Data":"2690f7e3d0fb44fc0e7ca85dcedf0279cac704acf4eaf9010fba5974aee3f6e9"} Jul 29 08:47:34.366488 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.365982 2582 scope.go:117] "RemoveContainer" containerID="cb013d8603076a07e06ceac8d82de6534619d617532283de60cd9c439b0e1869" Jul 29 08:47:34.369093 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.369066 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" event={"ID":"e7d86643-8041-470c-b991-d3d51d2790a9","Type":"ContainerDied","Data":"7dd252aacafbb7ce57a3620eeee325b0c3bade732fab76ba69e7e5b6ac5b3642"} Jul 29 08:47:34.369192 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.369115 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn" Jul 29 08:47:34.379671 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.376822 2582 scope.go:117] "RemoveContainer" containerID="721bda34f1bf35e7c13c04cb98eab5b67e3f01cfc31816549b9e0c424525d306" Jul 29 08:47:34.388534 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.388505 2582 scope.go:117] "RemoveContainer" containerID="949b8871071d85eaabe0c00dc4ec8cd409c5cf343e17bc76d6107f3b4ee79556" Jul 29 08:47:34.390869 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.390842 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp"] Jul 29 08:47:34.393098 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.393077 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-klfdp"] Jul 29 08:47:34.397949 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.397928 2582 scope.go:117] "RemoveContainer" containerID="6ac19838c716a9755bf78d9e4651aa7810373c5aff3f020e3018f6d5d83f221e" Jul 29 08:47:34.402328 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.402303 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn"] Jul 29 08:47:34.405558 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.405530 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-2hvfn"] Jul 29 08:47:34.407373 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.407354 2582 scope.go:117] "RemoveContainer" containerID="7e1221619acf59c83ccec30dec6de83a4ee85a8b2da16785697cb1799c827cda" Jul 29 08:47:34.415372 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.415352 2582 scope.go:117] "RemoveContainer" containerID="a6ed2e0fe91cb9771cb999e9a56a5bc79520fa4ea60439c9c60b2e9cd900a2cf" Jul 29 08:47:34.423085 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.423066 2582 scope.go:117] "RemoveContainer" containerID="9c5d48007db3eea9f41e86743c5c6408841381093edb858199d5bd393eeda72e" Jul 29 08:47:34.431393 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.431375 2582 scope.go:117] "RemoveContainer" containerID="b634fa4280525edafed5989b9c234ad2a183a8df5acf71fbff98d75c8006467b" Jul 29 08:47:34.439663 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.439638 2582 scope.go:117] "RemoveContainer" containerID="d083839a23dddb3b9d51ae49dc6ab32d8e54f6d58c0dcc87edc4545efd25915e" Jul 29 08:47:34.447933 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.447913 2582 scope.go:117] "RemoveContainer" containerID="219c55c3cab4fad4702a461c322feeca9fb5692909c3c4d1d83fce449af8745e" Jul 29 08:47:34.455768 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:34.455747 2582 scope.go:117] "RemoveContainer" containerID="d5bceac6f46ca6a815bd6f41fb61ea0b8b096fe40501f6acf5ca00120a74125e" Jul 29 08:47:36.251008 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:36.250975 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" path="/var/lib/kubelet/pods/b47f876b-f8d0-497e-89fa-58b83b619157/volumes" Jul 29 08:47:36.251564 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:36.251550 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" path="/var/lib/kubelet/pods/e7d86643-8041-470c-b991-d3d51d2790a9/volumes" Jul 29 08:47:46.195789 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:46.195754 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:46.337278 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:46.337227 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:52.091680 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:52.091579 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:52.168358 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:52.168313 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:53.219301 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219267 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t25l6/must-gather-m9tq2"] Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219577 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="offline" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219588 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="offline" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219597 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="ui" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219601 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="ui" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219609 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="online" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219615 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="online" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219643 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="registry" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219649 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="registry" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219657 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="online" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219663 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="online" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219669 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="offline" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219674 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="offline" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219680 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="feast-init" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219685 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="feast-init" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219700 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="ui" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219705 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="ui" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219713 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="feast-apply" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219717 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="feast-apply" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219728 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="feast-init" Jul 29 08:47:53.219728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219732 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="feast-init" Jul 29 08:47:53.220375 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219741 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="feast-apply" Jul 29 08:47:53.220375 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219747 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="feast-apply" Jul 29 08:47:53.220375 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219796 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="offline" Jul 29 08:47:53.220375 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219802 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="online" Jul 29 08:47:53.220375 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219808 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="ui" Jul 29 08:47:53.220375 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219816 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="registry" Jul 29 08:47:53.220375 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219821 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="ui" Jul 29 08:47:53.220375 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219827 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47f876b-f8d0-497e-89fa-58b83b619157" containerName="online" Jul 29 08:47:53.220375 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.219833 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d86643-8041-470c-b991-d3d51d2790a9" containerName="offline" Jul 29 08:47:53.224445 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.224424 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:47:53.226728 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.226701 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t25l6\"/\"openshift-service-ca.crt\"" Jul 29 08:47:53.227254 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.227235 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t25l6\"/\"default-dockercfg-2kd6h\"" Jul 29 08:47:53.227365 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.227265 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t25l6\"/\"kube-root-ca.crt\"" Jul 29 08:47:53.230674 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.230649 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t25l6/must-gather-m9tq2"] Jul 29 08:47:53.233354 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.233331 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:53.238433 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.238411 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rhhw\" (UniqueName: \"kubernetes.io/projected/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-kube-api-access-5rhhw\") pod \"must-gather-m9tq2\" (UID: \"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4\") " pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:47:53.238531 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.238458 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-must-gather-output\") pod \"must-gather-m9tq2\" (UID: \"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4\") " pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:47:53.334573 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.334526 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:47:53.339664 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.339610 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhhw\" (UniqueName: \"kubernetes.io/projected/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-kube-api-access-5rhhw\") pod \"must-gather-m9tq2\" (UID: \"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4\") " pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:47:53.339746 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.339698 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-must-gather-output\") pod \"must-gather-m9tq2\" (UID: \"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4\") " pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:47:53.340021 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.340003 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-must-gather-output\") pod \"must-gather-m9tq2\" (UID: \"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4\") " pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:47:53.350241 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.350211 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rhhw\" (UniqueName: \"kubernetes.io/projected/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-kube-api-access-5rhhw\") pod \"must-gather-m9tq2\" (UID: \"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4\") " pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:47:53.545470 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.545360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:47:53.675141 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.675090 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t25l6/must-gather-m9tq2"] Jul 29 08:47:53.677434 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:47:53.677395 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eaa685d_3a9d_41aa_8dfe_d1159f82b5b4.slice/crio-607f2fc1b940cffad780bfd62ca1d1e4b9d7716ceed67f57d8a2285363f20a17 WatchSource:0}: Error finding container 607f2fc1b940cffad780bfd62ca1d1e4b9d7716ceed67f57d8a2285363f20a17: Status 404 returned error can't find the container with id 607f2fc1b940cffad780bfd62ca1d1e4b9d7716ceed67f57d8a2285363f20a17 Jul 29 08:47:53.679351 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:53.679332 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:47:54.456502 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:47:54.456427 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t25l6/must-gather-m9tq2" event={"ID":"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4","Type":"ContainerStarted","Data":"607f2fc1b940cffad780bfd62ca1d1e4b9d7716ceed67f57d8a2285363f20a17"} Jul 29 08:48:00.485726 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:00.485686 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t25l6/must-gather-m9tq2" event={"ID":"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4","Type":"ContainerStarted","Data":"4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a"} Jul 29 08:48:00.485726 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:00.485734 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t25l6/must-gather-m9tq2" event={"ID":"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4","Type":"ContainerStarted","Data":"8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872"} Jul 29 08:48:00.515224 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:00.515163 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t25l6/must-gather-m9tq2" podStartSLOduration=1.273752643 podStartE2EDuration="7.515147142s" totalImagesPullingTime="6.241394499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:00.514348751 +0000 UTC m=+1800.827529612" watchObservedRunningTime="2026-07-29 08:48:00.515147142 +0000 UTC m=+1800.828328039" Jul 29 08:48:10.528529 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:10.528497 2582 generic.go:356] "Generic (PLEG): container finished" podID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" containerID="8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872" exitCode=0 Jul 29 08:48:10.528993 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:10.528572 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t25l6/must-gather-m9tq2" event={"ID":"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4","Type":"ContainerDied","Data":"8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872"} Jul 29 08:48:10.528993 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:10.528958 2582 scope.go:117] "RemoveContainer" containerID="8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872" Jul 29 08:48:10.712263 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:10.712232 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-t25l6_must-gather-m9tq2_7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4/gather/0.log" Jul 29 08:48:11.178016 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:11.177989 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:48:11.427349 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:11.427315 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:48:11.491800 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:11.491768 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:48:14.325961 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:14.325920 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mcpcq_ddbd8f37-c168-4e7a-b06a-9bfdd9579c7f/global-pull-secret-syncer/0.log" Jul 29 08:48:14.439039 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:14.439003 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" Jul 29 08:48:14.557363 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:14.557329 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-90.ec2.internal_a35ab5c768d8e59545c7458cdd8d9519/haproxy/0.log" Jul 29 08:48:16.153776 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.153739 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-t25l6/must-gather-m9tq2"] Jul 29 08:48:16.154299 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.154193 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-t25l6/must-gather-m9tq2" podUID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" containerName="copy" containerID="cri-o://4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a" gracePeriod=2 Jul 29 08:48:16.155707 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.155678 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-t25l6/must-gather-m9tq2"] Jul 29 08:48:16.156083 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:48:16.156060 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-m9tq2\" is forbidden: User \"system:node:ip-10-0-131-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-t25l6\": no relationship found between node 'ip-10-0-131-90.ec2.internal' and this object" podUID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:48:16.385612 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.385593 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-t25l6_must-gather-m9tq2_7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4/copy/0.log" Jul 29 08:48:16.387225 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.387210 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:48:16.451985 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.451886 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rhhw\" (UniqueName: \"kubernetes.io/projected/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-kube-api-access-5rhhw\") pod \"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4\" (UID: \"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4\") " Jul 29 08:48:16.451985 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.451925 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-must-gather-output\") pod \"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4\" (UID: \"7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4\") " Jul 29 08:48:16.452354 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.452334 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" (UID: "7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:48:16.454322 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.454297 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-kube-api-access-5rhhw" (OuterVolumeSpecName: "kube-api-access-5rhhw") pod "7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" (UID: "7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4"). InnerVolumeSpecName "kube-api-access-5rhhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:48:16.552158 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.552129 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-t25l6_must-gather-m9tq2_7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4/copy/0.log" Jul 29 08:48:16.552479 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.552458 2582 generic.go:356] "Generic (PLEG): container finished" podID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" containerID="4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a" exitCode=143 Jul 29 08:48:16.552552 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.552504 2582 scope.go:117] "RemoveContainer" containerID="4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a" Jul 29 08:48:16.552552 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.552512 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t25l6/must-gather-m9tq2" Jul 29 08:48:16.552666 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.552547 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rhhw\" (UniqueName: \"kubernetes.io/projected/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-kube-api-access-5rhhw\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:48:16.552666 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.552570 2582 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4-must-gather-output\") on node \"ip-10-0-131-90.ec2.internal\" DevicePath \"\"" Jul 29 08:48:16.561338 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.561309 2582 scope.go:117] "RemoveContainer" containerID="8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872" Jul 29 08:48:16.575866 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.575847 2582 scope.go:117] "RemoveContainer" containerID="4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a" Jul 29 08:48:16.576126 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:48:16.576104 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a\": container with ID starting with 4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a not found: ID does not exist" containerID="4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a" Jul 29 08:48:16.576198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.576139 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a"} err="failed to get container status \"4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a\": rpc error: code = NotFound desc = could not find container \"4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a\": container with ID starting with 4bdf8cf07389d1a273aa41afbc2987516258825e2afe1cd6ae85c9a73ba08b0a not found: ID does not exist" Jul 29 08:48:16.576198 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.576166 2582 scope.go:117] "RemoveContainer" containerID="8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872" Jul 29 08:48:16.576412 ip-10-0-131-90 kubenswrapper[2582]: E0729 08:48:16.576395 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872\": container with ID starting with 8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872 not found: ID does not exist" containerID="8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872" Jul 29 08:48:16.576473 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.576457 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872"} err="failed to get container status \"8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872\": rpc error: code = NotFound desc = could not find container \"8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872\": container with ID starting with 8e948b8d4a12d76c1afcaab25f23f657f93fe5283b1cfb8fc3e1a3fcff3aa872 not found: ID does not exist" Jul 29 08:48:16.802027 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.801995 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:48:16.863033 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:16.863003 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gd25w_f2d9ce03-0925-43ae-9a2a-f4a6bb186df0/konnectivity-agent/0.log" limit=1024 Jul 29 08:48:18.057212 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.057173 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c554ac19-c275-480a-9b1f-c526717e33c5/alertmanager/0.log" Jul 29 08:48:18.088216 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.088191 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c554ac19-c275-480a-9b1f-c526717e33c5/config-reloader/0.log" Jul 29 08:48:18.120242 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.120212 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c554ac19-c275-480a-9b1f-c526717e33c5/kube-rbac-proxy-web/0.log" Jul 29 08:48:18.147608 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.147585 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c554ac19-c275-480a-9b1f-c526717e33c5/kube-rbac-proxy/0.log" Jul 29 08:48:18.176599 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.176577 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c554ac19-c275-480a-9b1f-c526717e33c5/kube-rbac-proxy-metric/0.log" Jul 29 08:48:18.202192 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.202163 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c554ac19-c275-480a-9b1f-c526717e33c5/prom-label-proxy/0.log" Jul 29 08:48:18.229132 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.229110 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c554ac19-c275-480a-9b1f-c526717e33c5/init-config-reloader/0.log" Jul 29 08:48:18.250920 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.250892 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" path="/var/lib/kubelet/pods/7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4/volumes" Jul 29 08:48:18.276462 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.276429 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-b2lj8_1aed42ac-58e1-4a5c-b9fb-7acc1f0c94ec/cluster-monitoring-operator/0.log" Jul 29 08:48:18.304450 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.304423 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-v6trg_c3b42f2e-0adf-469f-9e88-befa1af691da/kube-state-metrics/0.log" Jul 29 08:48:18.336097 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.336021 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-v6trg_c3b42f2e-0adf-469f-9e88-befa1af691da/kube-rbac-proxy-main/0.log" Jul 29 08:48:18.363634 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.363593 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-v6trg_c3b42f2e-0adf-469f-9e88-befa1af691da/kube-rbac-proxy-self/0.log" Jul 29 08:48:18.398339 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.398314 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5848859c-49qsj_86b214eb-198b-4652-aac4-515feb24caef/metrics-server/0.log" Jul 29 08:48:18.636484 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.636404 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kl5pl_847664e3-1d31-418d-b9d4-85c12c854a24/node-exporter/0.log" Jul 29 08:48:18.660239 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.660215 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kl5pl_847664e3-1d31-418d-b9d4-85c12c854a24/kube-rbac-proxy/0.log" Jul 29 08:48:18.684989 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:18.684963 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kl5pl_847664e3-1d31-418d-b9d4-85c12c854a24/init-textfile/0.log" Jul 29 08:48:19.057072 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:19.057035 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-5dnwd_f5d8e82e-0f59-4605-8151-3aeaee89fe3e/prometheus-operator-admission-webhook/0.log" Jul 29 08:48:19.089861 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:19.089831 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b44fd54c6-n55ng_95ad9287-997b-4533-8ac7-7b063a4a1664/telemeter-client/0.log" Jul 29 08:48:19.112025 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:19.112003 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b44fd54c6-n55ng_95ad9287-997b-4533-8ac7-7b063a4a1664/reload/0.log" Jul 29 08:48:19.145735 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:19.145715 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b44fd54c6-n55ng_95ad9287-997b-4533-8ac7-7b063a4a1664/kube-rbac-proxy/0.log" Jul 29 08:48:20.542787 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:20.542758 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-5fc4q_33011701-b1db-40f6-879e-76e769b46374/networking-console-plugin/0.log" Jul 29 08:48:21.435259 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.435223 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-648d8678b4-dg7bw_4447f42f-6753-4a4e-8f01-5f57a64b5c75/console/0.log" Jul 29 08:48:21.468535 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.468505 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-xwpsg_bf3a040b-6446-4f34-8d64-da906783e51e/download-server/0.log" Jul 29 08:48:21.785638 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.785582 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9"] Jul 29 08:48:21.786059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.785941 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" containerName="gather" Jul 29 08:48:21.786059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.785952 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" containerName="gather" Jul 29 08:48:21.786059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.785962 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" containerName="copy" Jul 29 08:48:21.786059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.785967 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" containerName="copy" Jul 29 08:48:21.786059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.786016 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" containerName="gather" Jul 29 08:48:21.786059 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.786026 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eaa685d-3a9d-41aa-8dfe-d1159f82b5b4" containerName="copy" Jul 29 08:48:21.789662 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.789641 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.792661 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.792639 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rrptk\"/\"openshift-service-ca.crt\"" Jul 29 08:48:21.792796 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.792714 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rrptk\"/\"kube-root-ca.crt\"" Jul 29 08:48:21.793307 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.793290 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rrptk\"/\"default-dockercfg-qljdj\"" Jul 29 08:48:21.798079 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.798058 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9"] Jul 29 08:48:21.897235 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.897193 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-podres\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.897428 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.897254 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txwng\" (UniqueName: \"kubernetes.io/projected/d2cac92e-99e1-44fb-ba51-a345b9829d16-kube-api-access-txwng\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.897428 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.897310 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-proc\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.897428 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.897361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-lib-modules\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.897428 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.897383 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-sys\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.998543 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.998497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-lib-modules\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.998543 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.998546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-sys\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.998821 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.998611 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-podres\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.998821 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.998702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txwng\" (UniqueName: \"kubernetes.io/projected/d2cac92e-99e1-44fb-ba51-a345b9829d16-kube-api-access-txwng\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.998821 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.998706 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-sys\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.998821 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.998726 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-podres\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.998821 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.998714 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-lib-modules\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.998821 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.998731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-proc\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:21.998821 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:21.998781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2cac92e-99e1-44fb-ba51-a345b9829d16-proc\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:22.007267 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:22.007246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txwng\" (UniqueName: \"kubernetes.io/projected/d2cac92e-99e1-44fb-ba51-a345b9829d16-kube-api-access-txwng\") pod \"perf-node-gather-daemonset-qpql9\" (UID: \"d2cac92e-99e1-44fb-ba51-a345b9829d16\") " pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:22.100060 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:22.099968 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:22.237027 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:22.237002 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9"] Jul 29 08:48:22.239738 ip-10-0-131-90 kubenswrapper[2582]: W0729 08:48:22.239705 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2cac92e_99e1_44fb_ba51_a345b9829d16.slice/crio-1d454257bdad16aee26d19323e0102b12d0e3eb6dedc5f2885a888095ed47f94 WatchSource:0}: Error finding container 1d454257bdad16aee26d19323e0102b12d0e3eb6dedc5f2885a888095ed47f94: Status 404 returned error can't find the container with id 1d454257bdad16aee26d19323e0102b12d0e3eb6dedc5f2885a888095ed47f94 Jul 29 08:48:22.578477 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:22.578432 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" event={"ID":"d2cac92e-99e1-44fb-ba51-a345b9829d16","Type":"ContainerStarted","Data":"bb4e4186f10dc37ccc7b36893c1b81891f25625eea9ecc00be9acb118de734d1"} Jul 29 08:48:22.578694 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:22.578501 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" event={"ID":"d2cac92e-99e1-44fb-ba51-a345b9829d16","Type":"ContainerStarted","Data":"1d454257bdad16aee26d19323e0102b12d0e3eb6dedc5f2885a888095ed47f94"} Jul 29 08:48:22.578694 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:22.578659 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:22.593695 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:22.593645 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" podStartSLOduration=1.593614181 podStartE2EDuration="1.593614181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:22.592958986 +0000 UTC m=+1822.906139845" watchObservedRunningTime="2026-07-29 08:48:22.593614181 +0000 UTC m=+1822.906795042" Jul 29 08:48:22.886282 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:22.886201 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kn46w_ba9d8639-d6b0-463a-bd27-e373d9e10b30/dns/0.log" Jul 29 08:48:22.909566 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:22.909534 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kn46w_ba9d8639-d6b0-463a-bd27-e373d9e10b30/kube-rbac-proxy/0.log" Jul 29 08:48:22.963222 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:22.963190 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cwbns_79567eb3-7610-4e72-b625-6700f758a78a/dns-node-resolver/0.log" Jul 29 08:48:23.413144 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:23.413116 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5vj99_8971e266-0ba5-4a9f-a516-83954d7c971a/node-ca/0.log" Jul 29 08:48:24.204762 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:24.204737 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-849c4d5579-zcssz_d95ac6b4-144f-44f3-841e-80421f1f7559/router/0.log" Jul 29 08:48:24.548211 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:24.548178 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g2cp2_8a970a63-ce77-4e38-8bbf-3c2a727e0b2f/serve-healthcheck-canary/0.log" Jul 29 08:48:25.054880 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:25.054849 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-86bbc_f2eec8d1-92dc-4480-8bd0-df9135ee6f8a/kube-rbac-proxy/0.log" Jul 29 08:48:25.085851 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:25.085822 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-86bbc_f2eec8d1-92dc-4480-8bd0-df9135ee6f8a/exporter/0.log" Jul 29 08:48:25.112561 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:25.112536 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-86bbc_f2eec8d1-92dc-4480-8bd0-df9135ee6f8a/extractor/0.log" Jul 29 08:48:28.592991 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:28.592959 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rrptk/perf-node-gather-daemonset-qpql9" Jul 29 08:48:30.063520 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:30.063495 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-p6pgj_ac99683b-c447-476c-af04-eee88294cbc1/migrator/0.log" Jul 29 08:48:30.089712 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:30.089684 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-p6pgj_ac99683b-c447-476c-af04-eee88294cbc1/graceful-termination/0.log" Jul 29 08:48:30.461890 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:30.461858 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh_4c589bb2-2f4a-448e-a03b-2d1579ce3f62/kube-storage-version-migrator-operator/1.log" Jul 29 08:48:30.462743 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:30.462728 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-p76bh_4c589bb2-2f4a-448e-a03b-2d1579ce3f62/kube-storage-version-migrator-operator/0.log" Jul 29 08:48:31.537423 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:31.537399 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j5cxr_d5c50b1e-53b1-4dab-87ed-2b246ce43f74/kube-multus-additional-cni-plugins/0.log" Jul 29 08:48:31.561726 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:31.561699 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j5cxr_d5c50b1e-53b1-4dab-87ed-2b246ce43f74/egress-router-binary-copy/0.log" Jul 29 08:48:31.585632 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:31.585594 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j5cxr_d5c50b1e-53b1-4dab-87ed-2b246ce43f74/cni-plugins/0.log" Jul 29 08:48:31.612123 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:31.612100 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j5cxr_d5c50b1e-53b1-4dab-87ed-2b246ce43f74/bond-cni-plugin/0.log" Jul 29 08:48:31.636663 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:31.636633 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j5cxr_d5c50b1e-53b1-4dab-87ed-2b246ce43f74/routeoverride-cni/0.log" Jul 29 08:48:31.662396 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:31.662371 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j5cxr_d5c50b1e-53b1-4dab-87ed-2b246ce43f74/whereabouts-cni-bincopy/0.log" Jul 29 08:48:31.694501 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:31.694476 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j5cxr_d5c50b1e-53b1-4dab-87ed-2b246ce43f74/whereabouts-cni/0.log" Jul 29 08:48:32.143895 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:32.143866 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g2sxx_4b9ccf77-56ae-4384-897d-44864edecb7b/kube-multus/0.log" Jul 29 08:48:32.207500 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:32.207471 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5pfnw_30908914-7c33-4a7b-832a-bbf63c72b09b/network-metrics-daemon/0.log" Jul 29 08:48:32.237837 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:32.237810 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5pfnw_30908914-7c33-4a7b-832a-bbf63c72b09b/kube-rbac-proxy/0.log" Jul 29 08:48:33.889402 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:33.889353 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sx2cm_4d066258-79b7-4768-b5f2-b11990e7d37e/ovn-controller/0.log" Jul 29 08:48:33.909446 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:33.909415 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sx2cm_4d066258-79b7-4768-b5f2-b11990e7d37e/ovn-acl-logging/0.log" Jul 29 08:48:33.939713 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:33.939674 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sx2cm_4d066258-79b7-4768-b5f2-b11990e7d37e/kube-rbac-proxy-node/0.log" Jul 29 08:48:33.964842 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:33.964809 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sx2cm_4d066258-79b7-4768-b5f2-b11990e7d37e/kube-rbac-proxy-ovn-metrics/0.log" Jul 29 08:48:33.991777 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:33.991751 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sx2cm_4d066258-79b7-4768-b5f2-b11990e7d37e/northd/0.log" Jul 29 08:48:34.019862 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:34.019841 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sx2cm_4d066258-79b7-4768-b5f2-b11990e7d37e/nbdb/0.log" Jul 29 08:48:34.048511 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:34.048479 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sx2cm_4d066258-79b7-4768-b5f2-b11990e7d37e/sbdb/0.log" Jul 29 08:48:34.114209 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:34.114176 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sx2cm_4d066258-79b7-4768-b5f2-b11990e7d37e/ovnkube-controller/0.log" Jul 29 08:48:35.212709 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:35.212676 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-c9tjk_c8a2ea3c-92cd-45a6-a55e-38ab4b60f2dd/network-check-target-container/0.log" Jul 29 08:48:36.162247 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:36.162219 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8sqqv_9fd6cf4a-6a4b-47d6-b111-f466cc1d1c63/iptables-alerter/0.log" Jul 29 08:48:36.833834 ip-10-0-131-90 kubenswrapper[2582]: I0729 08:48:36.833804 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-585dp_b96ceeee-07bd-4a3a-9792-149aacc416b6/tuned/0.log"